Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a completely managed, extremely accessible, and safe Apache Kafka service. Amazon MSK reduces the work wanted to arrange, scale, and handle Apache Kafka in manufacturing. With Amazon MSK, you may create a cluster in minutes and begin sending information.
With Amazon MSK Serverless, you may run Apache Kafka with out having to handle the underlying infrastructure. Amazon MSK will robotically provision, scale, and handle your Apache Kafka clusters, so you may focus in your functions with out worrying concerning the operational overhead. Moreover, MSK Serverless affords fine-grained, pay-as-you-go pricing, making it an economical choice for organizations with unpredictable workloads.
Connecting to MSK Serverless is simple. You may arrange a serverless cluster utilizing the API or AWS Administration Console in minutes. MSK Serverless supplies bootstrap data as a personal DNS endpoint, permitting shoppers to connect with the serverless Apache Kafka cluster. A typical use case of utilizing MSK Serverless is an on-premises consumer that should course of real-time information streams. Nevertheless, the personal DNS endpoint is just accessible from digital personal clouds (VPCs) which have been configured to attach and isn’t immediately resolvable from an on-premises community. This may pose a problem for on-premises shoppers to find and connect with the MSK Serverless cluster. On this publish, we information you thru a step-by-step course of to attach your on-premises consumer to MSK Serverless, overcoming this problem.
Resolution overview
The next diagram illustrates the answer structure.
The circulation of the answer is as follows:
- The DNS question on your MSK endpoint is routed to a regionally configured on-premises DNS server.
- The on-premises DNS as configured performs conditional forwarding for
kafka-serverless.REPLACE-MSK-SERVERLESS-REGION.amazonaws.comto an Amazon Route 53 inbound resolver endpoint IP handle. - The inbound resolver endpoint performs DNS decision by forwarding the question to the personal hosted zone that was created together with the MSK Serverless cluster.
- The IP addresses returned by the DNS question are the personal IP addresses of the interface VPC endpoint, which permit your on-premises host to ascertain personal connectivity over AWS VPN or AWS Direct Join.
- The interface endpoint is a set of a number of elastic community interfaces with a personal IP handle in your account that serves as an entry level for site visitors destined to a MSK Serverless service.
Observe that at the moment, this answer works just for MSK Serverless clusters with a single VPC.
Conditions
On this part, we talk about the prerequisite steps to finish with the intention to implement this answer.
Set up community connectivity between on premises and the AWS Cloud
To make use of MSK Serverless out of your on-premises community, it’s essential set up a community connection between your on-premises surroundings and the VPC that you’ve got arrange for MSK Serverless. Numerous safe strategies can be found to attach your on-premises community to the AWS Cloud. Seek advice from Community-to-Amazon VPC connectivity choices for extra data.
Create a safety group for permitting inbound TCP/UDP connections out of your on-premises community
Create a safety group with the next configurations on the identical VPC that you simply configured for MSK Serverless:
Inbound rule:
- Supply: [On-premises CIDR range]
- Protocol: TCP/UDP
- Port Vary: 53
Outbound rule: Go away it to default
For extra data, confer with Work with safety teams.
Replace the MSK safety group for inbound connections out of your on-premises community
To make sure that your MSK Serverless cluster may be accessed out of your on-premises community, it’s essential regulate the cluster’s safety group settings to permit incoming site visitors out of your community on TCP port 9098. Full the next steps:
- On the Amazon MSK console, select Clusters within the navigation pane.
- Navigate to your serverless MSK cluster’s properties.
- Select the safety group related together with your MSK cluster.
As a result of MSK Serverless helps configuring a number of VPCs, ensure that to decide on the safety group related to the VPC that you simply configured for connecting out of your on-premises community.
- To allow connections out of your on-premises CIDR block to MSK Serverless, add an inbound rule that enables site visitors on TCP port 9098 out of your on-premises CIDR.
This ensures that your on-premises community can talk with MSK Serverless on the desired port.
Configure a Route 53 inbound resolver endpoint
MSK Serverless supplies a DNS endpoint that serves as the place to begin for an Apache Kafka consumer to connect with the cluster. Nevertheless, this endpoint isn’t publicly discoverable and might solely be accessed from inside the configured VPC. To resolve the serverless DNS endpoint exterior of your VPC, you may arrange a Route 53 resolver endpoint. This lets you entry the endpoint securely by making a hybrid cloud setup over VPN or Direct Join.
To configure the Route 53 resolver utilizing the console, full the next steps:
- On the Route 53 console, beneath Resolver within the navigation pane, select Inbound endpoints.
- Select Create inbound endpoint.
- For Endpoint identify, enter the endpoint identify.
- For VPC within the Area, select the VPC the place you configured MSK Serverless.
- For Safety group for this endpoint, select the safety group that you simply created as a prerequisite for inbound TCP/UDP connections.
The safety group of the inbound resolver endpoint ought to permit site visitors from the on-premises DNS Server IP handle on TCP/UDP port 53.
Within the subsequent step, you add your IP addresses, guaranteeing that the variety of IP addresses matches the variety of subnets in your MSK cluster.
- Select the Availability Zones and subnets which might be the identical as your MSK Serverless community configuration.
- Choose Use an IP handle that’s chosen robotically.
- Select Create inbound endpoint.
- Copy the inbound endpoint IP addresses.
Configure the on-premises DNS server
On this instance, we use a Microsoft DNS server. To configure a conditional forwarder, full the next steps:
- Open DNS Supervisor.
- Run the next command within the Run command window:
- Select (right-click) Conditional Forwarders beneath the server of your selecting, then select New Conditional Forwarder.
Within the subsequent step, you enter kafka-serverless.REPLACE-MSK-SERVERLESS-REGION.amazonaws.com, utilizing the IP handle of Route 53 inbound resolver endpoints that you simply created earlier. You will discover the MSK endpoint data by accessing the cluster’s consumer data. To be taught extra about getting consumer data, confer with Getting the bootstrap brokers for an Amazon MSK cluster.
- For DNS Area, enter your endpoint identify. For instance,
kafka-serverless.ap-southeast-2.amazonaws.com. Don’t enter the complete endpoint identify. - Select OK.
Check the DNS decision
DNS (Area Identify System) makes use of TCP/UDP port 53. To check whether or not you may join any of the Route 53 inbound endpoints, run the next command out of your on-premises consumer:
For instance: telnet 10.1.0.133 53
The next is a pattern output:
Run the next command to examine whether or not you may join with the MSK Serverless endpoint out of your on-premises consumer. To get the MSK Serverless endpoint data, confer with Create an MSK Serverless cluster.
For instance: dig boot-abcdc9.c3.kafka-serverless.ap-southeast-2.amazonaws.com +quick
The next is a pattern output:
If the DNS decision fails, examine your community connectivity from on premises. For extra details about troubleshooting connectivity points, confer with How do I troubleshoot VPN tunnel connectivity to an Amazon VPC or Troubleshooting AWS Direct Join.
After you create a serverless MSK cluster, the service robotically creates an interface VPC endpoint for the cluster. You should use the dig command as proven above to retrieve the VPC endpoint ID and its related IP handle, which confirms that you’re now ready to connect with the MSK Serverless cluster out of your on-premises surroundings.
Check your Kafka consumer
When you full the configuration of the Route 53 inbound resolver endpoint and on-premises DNS server, you may check your Kafka consumer from an on-premises community. For directions, confer with Create a consumer machine. This documentation guides you thru the mandatory steps to arrange your consumer machine and confirm that it could possibly efficiently connect with your MSK cluster out of your on-premises community.
Conclusion
MSK Serverless makes it simple so that you can handle your information. You don’t have to fret about organising and operating your individual Kafka cluster, which saves effort and time. On this publish, we explored the choice of on-premises connectivity with MSK Serverless and the way it can drastically profit organizations. By establishing this connection, you may acquire entry to a variety of real-time analytics use case potentialities and unlock the complete potential of your information.
We encourage you to attempt on-premises connectivity with MSK serverless.
Concerning the Authors
Masudur Rahaman Sayem is a Streaming Information Architect at AWS. He works with AWS prospects globally to design and construct information streaming architectures to unravel real-world enterprise issues. He makes a speciality of optimizing options that use streaming information companies and NoSQL. Sayem may be very captivated with distributed computing.
Akeef Khan is a Options Architect at Amazon Net Companies. He helps SMB Greenfield prospects undertake the cloud. While being a generalist SA, Akeef is captivated with networking.
