Tuesday, November 29, 2022
HomeBig DataQuestion and visualize Amazon Redshift operational metrics utilizing the Amazon Redshift plugin...

Question and visualize Amazon Redshift operational metrics utilizing the Amazon Redshift plugin for Grafana


Grafana is a wealthy interactive open-source device by Grafana Labs for visualizing knowledge throughout one or many knowledge sources. It’s utilized in quite a lot of trendy monitoring stacks, permitting you to have a standard technical base and apply frequent monitoring practices throughout totally different programs. Amazon Managed Grafana is a totally managed, scalable, and safe Grafana-as-a-service resolution developed by AWS in collaboration with Grafana Labs.

Amazon Redshift is probably the most broadly used knowledge warehouse within the cloud. You possibly can view your Amazon Redshift cluster’s operational metrics on the Amazon Redshift console, use AWS CloudWatch, and question Amazon Redshift system tables instantly out of your cluster. The primary two choices present a set of predefined common metrics and visualizations. The final one means that you can use the flexibleness of SQL to get deep insights into the main points of the workload. Nonetheless, querying system tables requires information of system desk buildings. To handle that, we got here up with a consolidated Amazon Redshift Grafana dashboard that visualizes a set of curated operational metrics and works on high of the Amazon Redshift Grafana knowledge supply. You possibly can simply add it to an Amazon Managed Grafana workspace, in addition to to every other Grafana deployments the place the info supply is put in.

This submit guides you thru a step-by-step course of to create an Amazon Managed Grafana workspace and configure an Amazon Redshift cluster with a Grafana knowledge supply for it. Lastly, we present you the right way to arrange the Amazon Redshift Grafana dashboard to visualise the cluster metrics.

Answer overview

The next diagram illustrates the answer structure.

The answer contains the next parts:

  • The Amazon Redshift cluster to get the metrics from.
  • Amazon Managed Grafana, with the Amazon Redshift knowledge supply plugin added to it. Amazon Managed Grafana communicates with the Amazon Redshift cluster through the Amazon Redshift Information Service API.
  • The Grafana internet UI, with the Amazon Redshift dashboard utilizing the Amazon Redshift cluster as the info supply. The online UI communicates with Amazon Managed Grafana through an HTTP API.

We stroll you thru the next steps through the configuration course of:

  1. Configure an Amazon Redshift cluster.
  2. Create a database consumer for Amazon Managed Grafana on the cluster.
  3. Configure a consumer in AWS Single Signal-On (AWS SSO) for Amazon Managed Grafana UI entry.
  4. Configure an Amazon Managed Grafana workspace and sign up to Grafana.
  5. Arrange Amazon Redshift as the info supply in Grafana.
  6. Import the Amazon Redshift dashboard provided with the info supply.

Stipulations

To comply with together with this walkthrough, it’s best to have the next conditions:

  • An AWS account
  • Familiarity with the essential ideas of the next providers:
    • Amazon Redshift
    • Amazon Managed Grafana
    • AWS SSO

Configure an Amazon Redshift cluster

For those who don’t have an Amazon Redshift cluster, create a pattern cluster earlier than continuing with the next steps. For this submit, we assume that the cluster identifier known as redshift-demo-cluster-1 and the admin consumer identify is awsuser.

  1. On the Amazon Redshift console, select Clusters within the navigation pane.
  2. Select your cluster.
  3. Select the Properties tab.

Redshift Cluster Properties

To make the cluster discoverable by Amazon Managed Grafana, you have to add a particular tag to it.

  1. Select Add tags. Redshift Cluster Tags
  2. For Key, enter GrafanaDataSource.
  3. For Worth, enter true.
  4. Select Save adjustments.

Redshift Cluster Tags

Create a database consumer for Amazon Managed Grafana

Grafana shall be instantly querying the cluster, and it requires a database consumer to connect with the cluster. On this step, we create the consumer redshift_data_api_user and apply some safety greatest practices.

  1. On the cluster particulars web page, select Question knowledge and Question in question editor v2.Query Editor v2
  2. Select the redshift-demo-cluster-1 cluster we created beforehand.
  3. For Database, enter the default dev.
  4. Enter the consumer identify and password that you just used to create the cluster.
  5. Select Create connection.Redshift SU
  6. Within the question editor, enter the next statements and select Run:
CREATE USER redshift_data_api_user PASSWORD '<password>' CREATEUSER;
ALTER USER redshift_data_api_user SET readonly TO TRUE;
ALTER USER redshift_data_api_user SET query_group TO 'superuser';

The primary assertion creates a consumer with superuser privileges essential to entry system tables and views (be certain to make use of a singular password). The second prohibits the consumer from making modifications. The final assertion isolates the queries the consumer can run to the superuser queue, in order that they don’t intervene with the principle workload.

On this instance, we use service managed permissions in Amazon Managed Grafana and a workspace AWS Id and Entry Administration (IAM) position as an authentication supplier within the Amazon Redshift Grafana knowledge supply. We create the database consumer redshift_data_api_user utilizing the AmazonGrafanaRedshiftAccess coverage.

Configure a consumer in AWS SSO for Amazon Managed Grafana UI entry

Two authentication strategies can be found for accessing Amazon Managed Grafana: AWS SSO and SAML. On this instance, we use AWS SSO.

  1. On the AWS SSO console, select Customers within the navigation pane.
  2. Select Add consumer.
  3. Within the Add consumer part, present the required data.

SSO add user

On this submit, we choose Ship an e mail to the consumer with password setup directions. You want to have the ability to entry the e-mail deal with you enter since you use this e mail additional within the course of.

  1. Select Subsequent to proceed to the subsequent step.
  2. Select Add consumer.

An e mail is shipped to the e-mail deal with you specified.

  1. Select Settle for invitation within the e mail.

You’re redirected to sign up as a brand new consumer and set a password for the consumer.

  1. Enter a brand new password and select Set new password to complete the consumer creation.

Configure an Amazon Managed Grafana workspace and sign up to Grafana

Now you’re able to arrange an Amazon Managed Grafana workspace.

  1. On the Amazon Grafana console, select Create workspace.
  2. For Workspace identify, enter a reputation, for instance grafana-demo-workspace-1.
  3. Select Subsequent.
  4. For Authentication entry, choose AWS Single Signal-On.
  5. For Permission sort, choose Service managed.
  6. Selected Subsequent to proceed.AMG Workspace configure
  7. For IAM permission entry settings, choose Present account.AMG permission
  8. For Information sources, choose Amazon Redshift.
  9. Select Subsequent to complete the workspace creation.Redshift to workspace

You’re redirected to the workspace web page.

Subsequent, we have to allow AWS SSO as an authentication methodology.

  1. On the workspace web page, select Assign new consumer or group.SSO new user
  2. Choose the beforehand created AWS SSO consumer beneath Customers and Choose customers and teams tables.SSO User

It is advisable to make the consumer an admin, as a result of we arrange the Amazon Redshift knowledge supply with it.

  1. Choose the consumer from the Customers record and select Make admin.
  2. Return to the workspace and select the Grafana workspace URL hyperlink to open the Grafana UI.AMG workspace
  3. Register with the consumer identify and password you created within the AWS SSO configuration step.

Arrange an Amazon Redshift knowledge supply in Grafana

To visualise the info in Grafana, we have to entry the info first. To take action, we should create an information supply pointing to the Amazon Redshift cluster.

  1. On the navigation bar, select the decrease AWS icon (there are two) after which select Redshift from the record.
  2. For Areas, select the Area of your cluster.
  3. Choose the cluster from the record and select Add 1 knowledge supply.Choose Redshift Cluster
  4. On the Provisioned knowledge sources web page, select Go to settings.
  5. For Identify, enter a reputation on your knowledge supply.
  6. By default, Authentication Supplier must be set as Workspace IAM Function, Default Area must be the Area of your cluster, and Cluster Identifier must be the identify of the chosen cluster.
  7. For Database, enter dev.
  8. For Database Person, enter redshift_data_api_user.
  9. Select Save & Take a look at.Settings for Data Source

A hit message ought to seem.

Data source working

Import the Amazon Redshift dashboard provided with the info supply

Because the final step, we import the default Amazon Redshift dashboard and guarantee that it really works.

  1. Within the knowledge supply we simply created, select Dashboards on the highest navigation bar and select Import to import the Amazon Redshift dashboard.Dashboards in the plugin
  2. Underneath Dashboards on the navigation sidebar, select Handle.
  3. Within the dashboards record, select Amazon Redshift.

The dashboard seem, exhibiting operational knowledge out of your cluster. Whenever you add extra clusters and create knowledge sources for them in Grafana, you may select them from the Information supply record on the dashboard.

Clear up

To keep away from incurring pointless expenses, delete the Amazon Redshift cluster, AWS SSO consumer, and Amazon Managed Grafana workspace assets that you just created as a part of this resolution.

Conclusion

On this submit, we lined the method of organising an Amazon Redshift dashboard working beneath Amazon Managed Grafana with AWS SSO authentication and querying from the Amazon Redshift cluster beneath the identical AWS account. This is only one option to create the dashboard. You possibly can modify the method to set it up with SAML as an authentication methodology, use customized IAM roles to handle permissions with extra granularity, question Amazon Redshift clusters outdoors of the AWS account the place the Grafana workspace is, use an entry key and secret or AWS Secrets and techniques Supervisor primarily based connection credentials in knowledge sources, and extra. You too can customise the dashboard by including or altering visualizations utilizing the feature-rich Grafana UI.

As a result of the Amazon Redshift knowledge supply plugin is an open-source venture, you may set up it in any Grafana deployment, whether or not it’s within the cloud, on premises, and even in a container working in your laptop computer. That means that you can seamlessly combine Amazon Redshift monitoring into just about all of your present Grafana-based monitoring stacks.

For extra particulars concerning the programs and processes described on this submit, seek advice from the next:


In regards to the Authors

Sergey Konoplev is a Senior Database Engineer on the Amazon Redshift group. Sergey has been specializing in automation and enchancment of database and knowledge operations for greater than a decade.

Milind Oke is a Information Warehouse Specialist Options Architect primarily based out of New York. He has been constructing knowledge warehouse options for over 15 years and makes a speciality of Amazon Redshift.

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments