Coder Social home page Coder Social logo

nri-mssql's Introduction

New Relic Open Source community plus project banner.

New Relic integration for Microsoft SQL Server

The New Relic integration for MS SQL Server captures critical performance metrics and inventory reported by a SQL Server Instance. Data on the SQL Server Instance and Databases is collected.

Inventory and metric data is collected via SQL queries to the Instance.

Configuration

A user with the necessary permissions to collect all the metrics and inventory can be configured as follows

USE master;
CREATE LOGIN newrelic WITH PASSWORD = 'tmppassword';
CREATE USER newrelic FOR LOGIN newrelic;
GRANT CONNECT SQL TO newrelic;
GRANT VIEW SERVER STATE TO newrelic;

-- Goes through each user database and adds public permissions
DECLARE @name NVARCHAR(max)
DECLARE db_cursor CURSOR FOR
SELECT NAME
FROM master.dbo.sysdatabases
WHERE NAME NOT IN ('master','msdb','tempdb','model')
OPEN db_cursor
FETCH NEXT FROM db_cursor INTO @name WHILE @@FETCH_STATUS = 0
BEGIN
	EXECUTE('USE "' + @name + '"; CREATE USER newrelic FOR LOGIN newrelic;' );
	FETCH next FROM db_cursor INTO @name
END
CLOSE db_cursor
DEALLOCATE db_cursor

Installation and usage

For installation and usage instructions, see our documentation web site.

Custom queries

To add custom queries, use the -custom_metrics_query option to provide a single query, or the -custom_metrics_config option to specify a YAML file with one or more queries, such as the sample mssql-custom-query.yml.sample

How attributes are named

Each query that returns a table of values will be parsed row by row, adding the MssqlCustomQuerySample event as follows:

  • The column name is the attribute name
  • Each row value in that column is the attribute value
  • The metric type is auto-detected whether it is a number (type GAUGE), or a string (type ATTRIBUTE)

One customizable attribute in each row can be configured by database values using the following names:

  • The column metric_name specifies its attribute name
  • The column metric_value specifies its attribute value
  • The column metric_type specifies its metric type, i.e. gauge or attribute

For example, the following query makes attributes named category_0, category_1, category_2 and so on.

SELECT CONCAT('category_', category_id) AS metric_name, name AS metric_value, category_type FROM syscategories

Specifying queries in YAML

When using a YAML file containing queries, you can specify the following parameters for each query:

  • query (required) contains the SQL query
  • database (optional) Prepends USE <database name>; to the SQL, and adds the database name as an attribute
  • prefix (optional) prefix to prepend to the attribute name
  • metric_name (optional) specify the name for the customizable attribute
  • metric_type (optional) specify the metric type for the customizable attribute

Compatibility

Check the official documentation website for compatibility and requirements.

Building

Golang is required to build the integration. We recommend Golang 1.11 or higher.

After cloning this repository, go to the directory of the MSSQL integration and build it:

$ make

The command above executes the tests for the MSSQL integration and builds an executable file called nri-mssql under the bin directory.

To start the integration, run nri-mssql:

$ ./bin/nri-mssql

If you want to know more about usage of ./bin/nri-mssql, pass the -help parameter:

$ ./bin/nri-mssql -help

Testing

To run the tests execute:

$ make test

Develop locally

To develop locally on M1 we need to leverage a different image, having few limitations and forward the port:

version: '3.1'
services:
  mssql:
    image: mcr.microsoft.com/azure-sql-edge
    ports:
      - "1433:1433"
    container_name: mssql
    environment:
      ACCEPT_EULA: Y
      SA_PASSWORD: secret123!
      MSSQL_PID: Developer
    restart: always

To connect with a msclient simply start the service:

$ docker-compose up
$ sqlcmd -S127.0.0.1 -USA -Psecret123! -q "SELECT * FROM sys.dm_os_performance_counters WHERE counter_name = 'Buffer cache hit ratio' or counter_name = 'Buffer cache hit ratio base'"

To install sqlcmd you could run:

$ brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
$ brew update
$ brew install mssql-tools

Obviously, you could also run the integration and leverage the debugger.

Support

Should you need assistance with New Relic products, you are in good hands with several support diagnostic tools and support channels.

New Relic offers NRDiag, a client-side diagnostic utility that automatically detects common problems with New Relic agents. If NRDiag detects a problem, it suggests troubleshooting steps. NRDiag can also automatically attach troubleshooting data to a New Relic Support ticket.

If the issue has been confirmed as a bug or is a Feature request, please file a Github issue.

Support Channels

Privacy

At New Relic we take your privacy and the security of your information seriously, and are committed to protecting your information. We must emphasize the importance of not sharing personal data in public forums, and ask all users to scrub logs and diagnostic information for sensitive information, whether personal, proprietary, or otherwise.

We define “Personal Data” as any information relating to an identified or identifiable individual, including, for example, your name, phone number, post code or zip code, Device ID, IP address, and email address.

For more information, review New Relic’s General Data Privacy Notice.

Contribute

We encourage your contributions to improve this project! Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.

If you have any questions, or to execute our corporate CLA (which is required if your contribution is on behalf of a company), drop us an email at [email protected].

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

If you would like to contribute to this project, review these guidelines.

To all contributors, we thank you! Without your contribution, this project would not be what it is today.

License

nri-mssql is licensed under the MIT License.

nri-mssql's People

Contributors

alejandrodnm avatar alvarocabanas avatar ardias avatar arvdias avatar camdencheek avatar carlossscastro avatar coreyarnold avatar cpheps avatar cristianciutea avatar davidbrota avatar davidgit avatar dependabot[bot] avatar fryckbos avatar gsanchezgavier avatar jfjoly avatar jsbnr avatar kang-makes avatar lchapman4 avatar maju6406 avatar marcsanmi avatar mariomac avatar matiasburni avatar newrelic-coreint-bot avatar paologallinaharbur avatar pnvnd avatar renovate[bot] avatar rhullah avatar sigilioso avatar tangollama avatar zackkendra avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nri-mssql's Issues

Use new configuration samples

Description

In some not identified cases the integration is getting hanged and since the configuration samples are based in the satandard configuration the Agent is not restarting the integration.

Expected Behavior

Use the new Confg style to laverage the timeout feature

NR Diag results

Steps to Reproduce

Not clear how to reproduce the root cause.

Your Environment

Additional context

Add Default Data scraping interval information

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Add Default Data scraping interval information to Readme

Feature Description

A clear and concise description of the feature you want or need.

Describe Alternatives

A clear and concise description of any alternative solutions or features you've considered. Are there examples you could link us to?

Additional context

Add any other context here.

Priority

Please help us better understand this feature request by choosing a priority from the following options:
[Nice to Have, Really Want, Must Have, Blocker]

Nice to Have

Document domain user configuration.

The integration supports Domain users but there is no documentation/examples on how to set up.
Add documentation and examples using Domain users to configure the integration.

Note: One of the know options is to leave the user and pass parameters empty.

Individual intervals for each custom query or possibility to only send custom metrics

Description

We have a huge amount of data within our SQL Server. Some queries can only be executed on a specific time interval. We cannot let the integration perform the same query each minute all the time. Some queries should only be executed on the below intervals as an example:

  • 60s
  • 120s (2m)
  • 300s (5m)
  • 3600s (1h)

Reason would be that some queries takes a lot of time to execute so we need to be able to specify the interval for each query. Otherwise it's like we're DDoS attacking our own servers with heavy queries.

The only workaround we have right now is to setup different integrations like below:

integrations:
  - name: nri-mssql
    env: 
      HOSTNAME: ST-AG-TEST
      USERNAME: HIDDEN
      PASSWORD: "HIDDEN"
      PORT: 1433
      CUSTOM_METRICS_CONFIG: 'C:\Program Files\New Relic\newrelic-infra\integrations.d\mssql-custom-query-2min.yml'
      ENABLE_BUFFER_METRICS: false
      ENABLE_DATABASE_RESERVE_METRICS: false
      METRICS: false
      INVENTORY: false
    inventory_source: config/mssql
    labels:
      purpose: MssqlCustomQuery
    interval: 120s

  - name: nri-mssql
    env: 
      HOSTNAME: ST-AG-TEST
      USERNAME: HIDDEN
      PASSWORD: "HIDDEN"
      PORT: 1433
      CUSTOM_METRICS_CONFIG: 'C:\Program Files\New Relic\newrelic-infra\integrations.d\mssql-custom-query-5min.yml'
      ENABLE_BUFFER_METRICS: false
      ENABLE_DATABASE_RESERVE_METRICS: false
      METRICS: false
      INVENTORY: false
    inventory_source: config/mssql
    labels:
      purpose: MssqlCustomQuery
    interval: 300s

To avoid duplicate data ingested by New Relic we have also created a drop rule by selecting our label like below:
SELECT * FROM MssqlInstanceSample, MssqlDatabaseSample WHERE label.purpose = 'MssqlCustomQuery'

So we have 3 integrations in total in our example. 1 main integration where we want the data sent to MssqlInstanceSample and MssqlDatabaseSample. The other two are sending duplicate data but we're dropping everything which isn't MssqlCustomQuerySample.

Acceptance Criteria

Possible to set individual interval for each custom query from the file specified within CUSTOM_METRICS_CONFIG.

Describe Alternatives

Another solution would be to have the possibility to disable data sent to MssqlInstanceSample and MssqlDatabaseSample or just have a setting to only enable MssqlCustomQuerySample for the specific integration used for custom queries. This would also help us a lot. We would like to do this on the server side to avoid unnecessary network traffic and unnecessary queries on our SQL Servers.

Dependencies

N/A

Additional context

N/A

Priority

[Nice to Have, Really Want, Must Have, Blocker]

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error ❌ Fail ⚠️ Warn ✅ Pass Ignored Total
0 0 0 7 0 7

Passed #

Click to see rules

license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

readme-starts-with-community-plus-header #

The first 5 lines contain all of the requested patterns. (README.md). The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/.

readme-contains-link-to-security-policy #

Contains a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

readme-contains-discuss-topic #

Contains a link to the appropriate discuss.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate discuss.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. All files passed this test.

third-party-notices-file-exists #

Found file (THIRD_PARTY_NOTICES.md). A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view.

Add insert_sql_max_length capability like is available on the java agent

Is your feature request related to a problem? Please describe.

On large database queries, the query text gets cut off due to buffer size limitations in the integration.

Feature Description

Looking for comparable configuration feature like what is provided in the Java APM agent. The agent is configurable in that you can override the default buffer size for query text (2000) by setting the insert_sql_max_length variable above 2000 length value in the newrelic.yml configuration file. See New Relic documentation
https://docs.newrelic.com/docs/apm/agents/java-agent/configuration/java-agent-configuration-config-file/

Describe Alternatives

Some other way to capture the entire text of large query statements

Additional context

Priority

Please help us better understand this feature request by choosing a priority from the following options:
Really Want

Int overflow for integrated metric instance.diskInBytes

Description

From the integration log:

Could not execute instance query: sql: Scan error on column index 0, name \"total_disk_space\": converting driver.Value type int64 (\"21339549696\") to a int: value out of range" integration_name=nri-mssql role=mssql runner_uid=e023e0872a

Taking a look at the source code, this is coming clearly from an int overflow here:

		query: `SELECT Sum(total_bytes) AS total_disk_space FROM (
			SELECT DISTINCT
			dovs.volume_mount_point,
			dovs.available_bytes available_bytes,
			dovs.total_bytes total_bytes
			FROM sys.master_files mf WITH (nolock)
			CROSS apply sys.Dm_os_volume_stats(mf.database_id, mf.file_id) dovs
			) drives`,
		dataModels: &[]struct {
			TotalDiskSpace *int `db:"total_disk_space" metric_name:"instance.diskInBytes" source_type:"gauge"`
		}{},

Expected Behavior

No error

Steps to Reproduce

Running this inside a windows container, where total disk space is 20GB (21339549696 bytes).

image

Your Environment

MSSQL Server 2022 CU12 inside a windows container.

Additional context

For Maintainers Only or Hero Triaging this bug

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error ❌ Fail ⚠️ Warn ✅ Pass Ignored Total
0 0 0 7 0 7

Passed #

Click to see rules

license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

readme-starts-with-community-plus-header #

The first 5 lines contain all of the requested patterns. (README.md). The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/.

readme-contains-link-to-security-policy #

Contains a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

readme-contains-discuss-topic #

Contains a link to the appropriate discuss.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate discuss.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. All files passed this test.

third-party-notices-file-exists #

Found file (THIRD_PARTY_NOTICES.md). A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view.

Allow/deny list for databases to be queried

Is your feature request related to a problem? Please describe.

I have servers that host ~thousands of databases. The OHI looks to query every database on the server
This can result in ~thousands of queries per minute from the OHI.

Feature Description

It'd be useful to be able to specify which databases should be queried, or which should be skipped, by any per-database queries.

I'd suggest, as a way to implement this, that the queries should only be attempted to be run on databases that the SQL login has been given permissions to. This would support an additional use case:

If there are any databases on a server with especially sensitive data, some users may not want to grant the agent's SQL login access to those databases.

Priority

Please help us better understand this feature request by choosing a priority from the following options:
[Nice to Have]

Add debug information when custom queries return empty result set.

When running custom queries we debug log errors that might be reported from running a query but we do not log anything when the query returns an empty result.
We should add such logging information to facilitate troubleshooting when no data is visible in NRDB but no errors are reported either.

Something along the lines "Query X returned 0 results"

https://github.com/newrelic/nri-mssql/blob/master/src/metrics/metrics.go#L140-L156

mssql-custom-query.yml cannot include inline comments in SQL query

Used one of your examples and anything after the inline comments was not included in the SQL query of a multi-line statement.

Description

I have been testing the multiple custom queries solution and used two queries. The second of which is from the examples file where I select the top 15 longest running queries. On line 168 of mssql-custom-query.yml.sample file you include an inline SQL comment to say that you are filtering out the custom query statement. Anything after this comment is ignored in the query as I added an additional clause to limit the results to a single database. My statement was as follows:

SELECT TOP 15
        @@SERVERNAME AS [sql_hostname],
        ISNULL(DB_NAME(t.dbid),'') AS [database_name],
        LEFT(t.[text], 50) AS [short_text], 
        qs.execution_count AS [execution_count],
        qs.total_worker_time AS [cpu_time_total_ms],
        qs.total_worker_time/qs.execution_count AS [cpu_time_avg_ms],     
        qs.total_physical_reads AS [physical_reads_total],
        qs.total_physical_reads/qs.execution_count AS [physical_reads_avg],
        qs.total_logical_reads AS [logical_reads_total],
        qs.total_logical_reads/qs.execution_count AS [logical_reads_avg],
        qs.total_logical_writes AS [logical_writes_total],
        qs.total_logical_writes/qs.execution_count AS [logical_writes_avg],
        qs.total_elapsed_time AS [duration_total_ms],
        qs.total_elapsed_time/qs.execution_count AS [duration_avg_ms],
        qs.creation_time AS [creation_time],
        t.[text] AS [complete_text]
      FROM sys.dm_exec_query_stats AS qs WITH (NOLOCK)
      CROSS APPLY sys.dm_exec_sql_text(plan_handle) AS t
      WHERE t.[text] NOT LIKE '%SELECT TOP 15%qs.execution_count%'        --Ignore this query
      AND ISNULL(DB_NAME(t.dbid),'') = 'MyDatabase' 
      ORDER BY qs.total_elapsed_time/qs.execution_count DESC 
      OPTION (RECOMPILE);

I captured the statement being used by the agent via SQL Profiler and this was the result.

SELECT TOP 15
  @@SERVERNAME AS [sql_hostname],
  ISNULL(DB_NAME(t.dbid),'') AS [database_name],
  LEFT(t.[text], 50) AS [short_text], 
  qs.execution_count AS [execution_count],
  qs.total_worker_time AS [cpu_time_total_ms],
  qs.total_worker_time/qs.execution_count AS [cpu_time_avg_ms],     
  qs.total_physical_reads AS [physical_reads_total],
  qs.total_physical_reads/qs.execution_count AS [physical_reads_avg],
  qs.total_logical_reads AS [logical_reads_total],
  qs.total_logical_reads/qs.execution_count AS [logical_reads_avg],
  qs.total_logical_writes AS [logical_writes_total],
  qs.total_logical_writes/qs.execution_count AS [logical_writes_avg],
  qs.total_elapsed_time AS [duration_total_ms],
  qs.total_elapsed_time/qs.execution_count AS [duration_avg_ms],
  qs.creation_time AS [creation_time],
  t.[text] AS [complete_text]
FROM sys.dm_exec_query_stats AS qs WITH (NOLOCK) CROSS APPLY sys.dm_exec_sql_text(plan_handle) AS t WHERE t.[text] NOT LIKE '%SELECT TOP 15%qs.execution_count%'

As you can see everything after the inline comment of "--Ignore this query" has been excluded including the order by and option to recompile.

Expected Behavior

I would have expected to see the entire custom statement being sent to SQL Server. Removing the comment resolved the issue, but as it's in the example file it is misleading.

Steps to Reproduce

  1. Include the custom query above including the comment in your mssql-custom-query.yml file and restart the agent.
  2. Open a SQL Profiler trace and include SQL:StatementCompleted and look at the statements generated by the nri-mssql agent and you will see any SQL that appears on a line after the inline comment will have been excluded.

Your Environment

Windows Server 2016 Standard
SQL Server 2019

Add File Watching for mssql-config

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
When I update the mssql-config.yml file, the agent does not automatically reflect this change. I have to restart the agent in order for the change to take effect.

Feature Description

A file watch should be added to nri-mssql, so that when mssql-config.yml updates with a new instance, it will automatically detect the change and implement this change.

Describe Alternatives

Currently I am working on a solution to automatically restart the pod when a config update occurs, but this is a bit of a work around.

Additional context

My environment is fairly dynamic and we have new sql server instances on a daily basis that need to automatically get added to the monitoring.

Priority

Nice to Have

Support Query Plans via custom query and push to NR Log API

Provide the ability to pull Query Plans from SQL Server via a custom query and push to New Relic's Log API.

The returned Query Plans can be very large, exceeding the limits of metrics or events, hence the use of the Log API with its much larger limits.

Windows authentication docs

Document windows authentication support according to this PR.
Add a test for these feature to support it and add documentation.

nri-mssql custom query returns error but does not show on info level logs

Hi,

When adding custom queries to the mssql integration and for some reason there is an error with the query or an error in SQL, in this case it was that the sql user does not have access to the table in the database to execute the query, SQL returns and error, but in the NewRelic logs there is no error shown at all but the data does not land up in NewRelic. This makes it difficult to see if something did go wrong, until you change the log level to debug, and then you see the error in the NewRelic logs.

Description

SQL user account that NewRelic agent uses to execute custom queries does not have rights on the database and this causes an error, but the error does not display in the NewRelic logs until debug on the logs are enabled.

Expected Behavior

All ERRORS from NewRelic related agents (msssql integration) should show up on the logs when logs are set on the appropriate level. When set to INFO all error, warn and info messages should show up in this log.

Troubleshooting or [NR Diag]

Log Line:
time="2023-10-18T11:13:55+02:00" level=debug msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-mssql line="[ERR] Could not execute custom query: mssql: The SELECT permission was denied on the object 'xxxx', database 'xxxx', schema 'xxxx'." runner_uid=xxxx server=xxxx

Note that this was on debug logging, when placed on info level logging this error does not show up and no error shows in the logs.

Steps to Reproduce

  1. Create a custom query for a table the new relic agent does not have permissions to
  2. Save the file and restart the agent
  3. Agent should try and execute the SQL query but without the correct privileges it should fail and NOT produce an error message when INFO logging is active.

Your Environment

NewRelic Infrastructure Agent 1.47.2
NRI-MSSQL agent 2.8.7

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error ❌ Fail ⚠️ Warn ✅ Pass Ignored Total
0 2 0 5 0 7

Fail #

readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

  • README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
    • 🔨 Suggested Fix: prepend [![Community Plus header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Community_Plus.png)](https://opensource.newrelic.com/oss-category/#community-plus) to file

code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Found files. Below is a list of files or patterns that failed:

  • CODE_OF_CONDUCT.md
    • 🔨 Suggested Fix: Remove file

Passed #

Click to see rules

license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

readme-contains-link-to-security-policy #

Contains a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

readme-contains-discuss-topic #

Contains a link to the appropriate discuss.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate discuss.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

third-party-notices-file-exists #

Found file (THIRD_PARTY_NOTICES.md). A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

docker-compose
tests/docker-compose.yml
  • mcr.microsoft.com/mssql/server 2017-latest
  • golang 1.22.2-bookworm
dockerfile
build/Dockerfile
  • golang 1.22.2-bookworm
github-actions
.github/workflows/automated_release.yaml
  • newrelic/coreint-automation v3
.github/workflows/on_prerelease.yaml
  • newrelic/coreint-automation v3
.github/workflows/on_push_pr.yaml
  • newrelic/coreint-automation v3
.github/workflows/on_release.yaml
  • newrelic/coreint-automation v3
.github/workflows/repolinter.yml
  • newrelic/coreint-automation v3
.github/workflows/security.yaml
  • newrelic/coreint-automation v3
gomod
go.mod
  • go 1.22.2
  • github.com/denisenkom/go-mssqldb v0.12.3
  • github.com/jmoiron/sqlx v1.4.0
  • github.com/newrelic/infra-integrations-sdk v3.8.2+incompatible
  • github.com/stretchr/testify v1.9.0
  • github.com/xeipuuv/gojsonschema v1.2.0
  • gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0
  • gopkg.in/yaml.v2 v2.4.0

  • Check this box to trigger a request for Renovate to run again on this repository

Revert disable upgrade package test

The upgrade tests for packages has been disabled for the very first release since the upgrade package is broken due to an old package in the repo.

After the first release revert the change #59

Have text fields in mssql-custom-query.yml.sample queries truncated to NRDB attribute limit (4095) to avoid log bloat.

Description

In some environments the complete_text of SQL queries reported by some of the custom queries in mssql-custom-query.yml.sample is longer than the NRDB limit. This causes warning messages in the agent log about the value being truncated, including the truncated text. This can lead to considerable log bloat.

Example log:
time="2024-04-18T10:09:31-04:00" level=warning msg="event truncated to NRDB limit" component=AgentContext entity_key="ms-instance:<REDACTED>:instance=<REDACTED>" length=129720 original="+map[displayName:<REDACTED> entityKey:ms-instance:EFP19-TR-SQL01:instance=efp19-tr-sql01 entityName:ms-instance:EFP19-TR-SQL01 eventType:MssqlCustomQuerySample event_type:MssqlCustomQuerySample host:EFP19-TR-SQL01 instance:<REDACTED> integrationName:com.newrelic.mssql integrationVersion:2.12.0 label.environment:Training label.product:<REDACTED> label.role:MSSQL longRunning_complete_text:CREATE PROCEDURE...<full text of query before truncation>

Acceptance Criteria

Update queries in mssql-custom-query.yml.sample so that text values do not exceed the NRDB attribute limit.

Describe Alternatives

Can work around the log bloat problem by either filtering out nri-mssql messages entirely per agent log config:
https://github.com/newrelic/infrastructure-agent/blob/master/docs/log_configuration.md#log-filters

Can also work around by editing the sample query and forcing the complete_text to be truncated to the NRDB limit when queried:
SELECT ...
LEFT(t.[text], 4095) AS [complete_text]
...

Dependencies

Infrastructure Agent
nri-mssql

NRDB limit checked by agent (section of the code that generates these log messages):
https://github.com/newrelic/infrastructure-agent/blob/db359de4f1042f5fd6b20276629f6b37989fe265/internal/agent/agent.go#L1157

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error ❌ Fail ⚠️ Warn ✅ Pass Ignored Total
0 0 0 7 0 7

Passed #

Click to see rules

license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

readme-starts-with-community-plus-header #

The first 5 lines contain all of the requested patterns. (README.md). The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/.

readme-contains-link-to-security-policy #

Contains a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

readme-contains-discuss-topic #

Contains a link to the appropriate discuss.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate discuss.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. All files passed this test.

third-party-notices-file-exists #

Found file (THIRD_PARTY_NOTICES.md). A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view.

MSSQL Integration - nri-mssql.exe not present in the expected directory when the agent is installed. Should work out of the box as documented in the examples in the integration documentation

NOTE: # The MSSQL integration appears to be looking for the nri-mssql executable in the wrong folder

Description

NOTE: # The Mssql integration was installed via the ms installer using default options. A CUSTOM_METRICS_QUERY was written to extract custom data.
TIP: #

Expected Behavior

NOTE: # The MSSQL integration should have worked out of the box.

NR Diag results

time="2022-02-21T15:52:13+02:00" level=debug msg="Integration name not found. Trying another folder, if any." component=integrations.Executables folder="C:\Program Files\New Relic\newrelic-infra\custom-integrations" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=debug msg="Error looking for integration executables in folder. Trying another folder, if any." component=integrations.Executables error="open C:\Program Files\New Relic\newrelic-infra\custom-integrations\bin: The system cannot find the file specified." folder="C:\Program Files\New Relic\newrelic-infra\custom-integrations\bin" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=debug msg="Integration name not found. Trying another folder, if any." component=integrations.Executables folder="C:\Program Files\New Relic\newrelic-infra\newrelic-integrations" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=debug msg="Integration name not found. Trying another folder, if any." component=integrations.Executables folder="C:\Program Files\New Relic\newrelic-infra\newrelic-integrations\bin" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=debug msg="Error looking for integration executables in folder. Trying another folder, if any." component=integrations.Executables error="open C:\Program Files\New Relic\newrelic-infra\bundled-plugins: The system cannot find the file specified." folder="C:\Program Files\New Relic\newrelic-infra\bundled-plugins" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=debug msg="Error looking for integration executables in folder. Trying another folder, if any." component=integrations.Executables error="open C:\Program Files\New Relic\newrelic-infra\bundled-plugins\bin: The system cannot find the path specified." folder="C:\Program Files\New Relic\newrelic-infra\bundled-plugins\bin" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=debug msg="Error looking for integration executables in folder. Trying another folder, if any." component=integrations.Executables error="open C:\Program Files\New Relic\newrelic-infra\plugins: The system cannot find the file specified." folder="C:\Program Files\New Relic\newrelic-infra\plugins" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=debug msg="Error looking for integration executables in folder. Trying another folder, if any." component=integrations.Executables error="open C:\Program Files\New Relic\newrelic-infra\plugins\bin: The system cannot find the path specified." folder="C:\Program Files\New Relic\newrelic-infra\plugins\bin" forName=mssql-server
time="2022-02-21T15:52:13+02:00" level=warning msg="can't instantiate integrations from file" component=integrations.Manager error="can't instantiate integration: can't find an executable given the name: mssql-server" file="C:\Program Files\New Relic\newrelic-infra\integrations.d\mssql-config.yml"

Steps to Reproduce

  • Download the MSSQL integration (https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/microsoft-sql-server-monitoring-integration/).

  • Run the MSI installer.

  • Create a mssql-config file with integration -name nri-mssql
    integrations:

    • name: nri-mssql
      env:
      METRICS: true
      HOSTNAME: localhost
      USERNAME: newrelic
      PASSWORD: ""

      Both port and instance can be omitted to use a default port of 1433

      PORT: 1433
      CUSTOM_METRICS_QUERY: >-
      SELECT 'count_entries' as metric_name,
      count(*) AS metric_value,
      'gauge' as metric_type
      FROM MyTable
      labels:
      env: production
      role: mssql
      customer: MyCustomer
  • Start the infrastructure agent service

  • Check the logs for the error
    level=debug msg="Integration name not found. Trying another folder, if any." component=integrations.Executables folder="C:\Program Files\New Relic\newrelic-infra\custom-integrations" forName=mssql-server

  • Stop the infrastructure agent

  • Then copy the newrelic-infra\newrelic-integrations\bin\nri-mssql.exe file to the newrelic-infra\custom-integrations\bin directory

  • Start the infra agent.

  • Check in the logs that the error has gone away.

TIP: # https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/microsoft-sql-server-monitoring-integration/

Your Environment

TIP: # This issue was encountered on Windows 10 Pro 10.0.18362 and Windows 2019

Additional context

TIP: # The nri-mssql executable had to be copied to the custom-integration\bin directory manually to get it to work.

Problems with MSSQL System Queries

I have a problem with the executable nri-mssql in that I need a change to be applied to the system queries this integration runs and when I rebuild the project to create the new exe, I am experiencing a problem with an error message when the integration runs:

This is the error message:

time="2020-07-31T14:50:38+02:00" level=error msg="Integration command failed" error="exit status 1" instance= integration=com.newrelic.mssql prefix=config/mssql stderr="[ERR] Error creating connection to SQL Server: InitialBytes InitializeSecurityContext failed 8009030c\n" working-dir="C:\Program Files\New Relic\newrelic-infra\newrelic-integrations"
time="2020-07-31T14:50:38+02:00" level=info msg="Integration health check finished with some errors" instance= integration=com.newrelic.mssql prefix=config/mssql working-dir="C:\Program Files\New Relic\newrelic-infra\newrelic-integrations"
time="2020-07-31T14:50:39+02:00" level=error msg="Integration command failed" error="exit status 1" instance= integration=com.newrelic.mssql prefix=config/mssql stderr="[ERR] Error creating connection to SQL Server: InitialBytes InitializeSecurityContext failed 8009030c\n" working-dir="C:\Program Files\New Relic\newrelic-infra\newrelic-integrations"

New Relic support are not engaged because this is a change to the original source.

This form is for nri-mysql bug reports and feature requests only.
This is NOT a help site. Do not ask help questions here.
If you need help, please use newrelic support.

This is the change I need applied in the instance.sql query:
Current query:

select
MAX(sys_mem.total_physical_memory_kb * 1024.0) AS total_physical_memory,
MAX(sys_mem.available_physical_memory_kb * 1024.0) AS available_physical_memory,
(Max(proc_mem.physical_memory_in_use_kb) / (Max(sys_mem.total_physical_memory_kb) * 1.0)) * 100 as memory_utilization
FROM sys.dm_os_process_memory proc_mem,
sys.dm_os_sys_memory sys_mem,
sys.dm_os_performance_counters perf_count WHERE object_name = 'SQLServer:Memory Manager'

Required query:

select
MAX(sys_mem.total_physical_memory_kb * 1024.0) AS total_physical_memory,
MAX(sys_mem.available_physical_memory_kb * 1024.0) AS available_physical_memory,
(Max(proc_mem.physical_memory_in_use_kb) / (Max(sys_mem.total_physical_memory_kb) * 1.0)) * 100 as memory_utilization
FROM sys.dm_os_process_memory proc_mem,
sys.dm_os_sys_memory sys_mem,
sys.dm_os_performance_counters perf_count WHERE object_name like '%Memory Manager%'

And the a new rebuild of the executable.

Docs about supported OSs for this integration are outdated

Description

We have an issue with the docs for this integration regarding the supported OSs. Our docs only list Windows as supported Os while we are releasing for all the OSs and architectures:

Screenshot 2022-07-11 at 12 57 55

Expected Behavior

We should update the docs to remove that part as there are no particularities on the OSs that we support for this integration, we simply support the same OSs that the agent support.

error while validating receipt of data for mssql-server-integration-installer: timed out

Description

Installation of MS SQL integration recipe is returning a failure message, although the integration seems to be installed properly and data is flowing to the New Relic dashboard.

{"level":"debug","msg":"installing recipe","name":"mssql-server-integration-installer","time":"2021-11-18T12:29:54-05:00"}
{"level":"debug","msg":"Found license key [redacted]********************************","time":"2021-11-18T12:29:54-05:00"}
{"level":"debug","msg":"preparing recipe","name":"mssql-server-integration-installer","time":"2021-11-18T12:29:54-05:00"}
{"error":"","guid":"","level":"debug","msg":"recipe event","recipe_name":"mssql-server-integration-installer","status":"INSTALLING","statusCount":2,"tasks":null,"time":"2021-11-18T12:29:54-05:00","validationDurationMs":0}
{"level":"debug","msg":"executing recipe mssql-server-integration-installer","time":"2021-11-18T12:29:55-05:00"}
{"level":"debug","msg":"skipping agent validation due to lack of validationUrl","time":"2021-11-18T12:30:14-05:00"}
{"level":"debug","msg":"Checking for data in New Relic (this may take a few minutes)...","time":"2021-11-18T12:30:14-05:00"}
{"level":"debug","msg":" Checking for data in New Relic (this may take a few minutes)...","time":"2021-11-18T12:35:14-05:00"}
{"error":"encountered an error while validating receipt of data for mssql-server-integration-installer: timed out waiting for validation to succeed","guid":"","level":"debug","msg":"recipe event","recipe_name":"mssql-server-integration-installer","status":"FAILED","statusCount":2,"tasks":null,"time":"2021-11-18T12:35:14-05:00","validationDurationMs":300006}
{"level":"debug","msg":"completed","time":"2021-11-18T12:35:15-05:00","timestamp":1637256915}
{"hasCanceledRecipes":false,"hasFailedRecipes":true,"hasInstalledRecipes":true,"hasSkippedRecipes":false,"level":"debug","msg":"final installation statuses updated","time":"2021-11-18T12:35:15-05:00"}
{"level":"debug","msg":"We encountered an issue during the installation: encountered an error while validating receipt of data for mssql-server-integration-installer: timed out waiting for validation to succeed.","time":"2021-11-18T12:35:15-05:00"}
{"level":"debug","msg":"If this problem persists, visit the documentation and support page for additional help here at https://one.newrelic.com/-/06vjAeZLKjP.","time":"2021-11-18T12:35:15-05:00"}

Expected Behavior

Validate connectivity and don't display an error.

NR Diag results

Attaching installer logs newrelic-cli.log (sanitized).

Steps to Reproduce

Install MS SQL on-host integration using the installer recipe mssql-server-integration-installer.

Your Environment

  • nri-mssql 2.8.1
  • Infrastructure agent 1.20.7
  • Microsoft Windows Server 2019 Datacenter 10.0.17763 Build 17763

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error ❌ Fail ⚠️ Warn ✅ Pass Ignored Total
0 0 0 7 0 7

Passed #

Click to see rules

license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

readme-starts-with-community-plus-header #

The first 5 lines contain all of the requested patterns. (README.md). The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/.

readme-contains-link-to-security-policy #

Contains a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

readme-contains-forum-topic #

Contains a link to the appropriate forum.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate forum.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. All files passed this test.

third-party-notices-file-exists #

Found file (THIRD_PARTY_NOTICES.md). A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view.

Incorrect entity names displayed for MSSQL RDS instances via On Host integration

Incorrect entity names displayed for MSSQL RDS instances via On Host integration

Description

Entity names of MSSQL RDS instances displayed in NR UI are different from the actual instance names.

Expected Behavior

Display the correct names of MSSQL RDS instances as configured

On Host integration config file

  • name: nri-mssql
    env:
    HOSTNAME: dev-ephesoft-db.cwkirvnl2kse.us-west-2.rds.amazonaws.com
    USERNAME:
    PASSWORD:
    PORT:
    ENABLE_SSL: False
    TRUST_SERVER_CERTIFICATE: True
    TIMEOUT: 0

Log

time="2023-04-14T17:22:55Z" level=debug msg="Received payload." component=integrations.emitter.Emitter env="map[ENABLE_BUFFER_METRICS:true ENABLE_DATABASE_RESERVE_METRICS:true ENABLE_SSL:False HOSTNAME:dev-ephesoft-db.cwkirvnl2kse.us-west-2.rds.amazonaws.com NRI_CONFIG_INTERVAL:15s PASSWORD: PORT:TIMEOUT:0 TRUST_SERVER_CERTIFICATE: USERNAME:]" integration_name=nri-mssql
time="2023-04-14T17:22:55Z" level=debug msg="Integration instances finished their execution. Waiting until next interval." component=integrations.runner.Runner env=production integration_name=nri-mssql role=mssql runner_uid=52cbaea0b9
time="2023-04-14T17:22:55Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender id="ms-instance:EC2AMAZ-5523GER:instance=ec2amaz-5523ger" numEvents=120 postCount=93720 timestamps="[2023-04-14 17:22:55 +0000 GMT]"
time="2023-04-14T17:22:55Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender id="ms-database:dev_ephesoft_tax_export_db:database=dev_ephesoft_tax_export_db:instance=ec2amaz-5523ger" numEvents=1 postCount=93720 timestamps="[2023-04-14 17:22:55 +0000 GMT]"
time="2023-04-14T17:22:55Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender id="ms-database:Ephesoft_App_db:database=ephesoft_app_db:instance=ec2amaz-5523ger" numEvents=1 postCount=93720 timestamps="[2023-04-14 17:22:55 +0000 GMT]"
time="2023-04-14T17:22:55Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender id="ms-database:Ephesoft_OM:database=ephesoft_om:instance=ec2amaz-5523ger" numEvents=1 postCount=93720 timestamps="[2023-04-14 17:22:55 +0000 GMT]"
time="2023-04-14T17:22:55Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender id="ms-database:Ephesoft_Report_Archive_db:database=ephesoft_report_archive_db:instance=ec2amaz-5523ger" numEvents=1 postCount=93720 timestamps="[2023-04-14 17:22:55 +0000 GMT]"
time="2023-04-14T17:22:55Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender id="ms-database:Ephesoft_Report_db:database=ephesoft_report_db:instance=ec2amaz-5523ger" numEvents=1 postCount=93720 timestamps="[2023-04-14 17:22:55 +0000 GMT]"
time="2023-04-14T17:22:55Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender id="ms-database:Eway:database=eway:instance=ec2amaz-5523ger" numEvents=1 postCount=93720 timestamps="[2023-04-14 17:22:55 +0000 GMT]"

Screenshot of entity in NR UI

image

The expected instance name dev-ephesoft-db but the displayed value is ec2amaz-5523ger

Permalink to entity

https://onenr.io/00jlpOvdBjW

Long Running Query - Buffer Collection

Hi,

We have been noticing buffer data queries taking nearly 2 minutes to complete with this query in the nri-mssql integration:

SELECT DB_NAME(database_id) AS db_name, COUNT_BIG() * (81024) AS buffer_pool_size
FROM sys.dm_os_buffer_descriptors WITH (NOLOCK) WHERE database_id <> 32767
AND DB_NAME(database_id) NOT IN ('master', 'tempdb', 'msdb', 'model', 'rdsadmin', 'distribution', 'model_msdb', 'model_replicatedmaster')
GROUP BY database_id

However, if we change the query we can make a huge performance impact:

SELECT DB_NAME(database_id) as 'db_name', buffer_pool_size* (81024) as 'buffer_pool_size'
FROM (
SELECT database_id, COUNT_BIG(
) AS buffer_pool_size
FROM sys.dm_os_buffer_descriptors a WITH (NOLOCK)
INNER JOIN sys.sysdatabases b WITH (NOLOCK) ON
b.dbid=a.database_id
WHERE b.dbid in
(
SELECT dbid from sys.sysdatabases WHERE name NOT IN ('master', 'tempdb', 'msdb', 'model', 'rdsadmin', 'distribution', 'model_msdb', 'model_replicatedmaster')
UNION ALL
SELECT 32767
)
GROUP BY database_id
) a

a trace shows the big difference in CPU (95% less), and Duration (80% less).

Would it be possible to investigate the performance impact of the query running in the default install of nri-mssql and amend accordingly?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.