AWS SDK ELASTICACHE AUTO DISCOVERY FOR GO

In this tutorial I’ll show you how to implement your own ElastiCache (Memcached) node Auto Discovery. Auto Discovery is useful when you need to add or remove nodes without having to manually add/remove the endpoints in your application. If in your application you’re only using one node your code would look like this:

mc := memcache.New("eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211")

If that memcached node began to evict items a new node would be needed. Adding a new node in AWS easy, go to your Cache Cluster console and click on the “Add Node” button. In a couple of minutes your node will up & running.

nodes

Since your application does not have any node Auto Discovery code you would need to add that new node manually:

mc := memcache.New("eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211", "eticket-cache.4xznmr.0002.euw1.cache.amazonaws.com:11211")

Then you’d build & upload your project to your server or servers. All these steps can be avoided by implementing node Auto Discovery.

The Code

package main

import (
   "fmt"
   "github.com/aws/aws-sdk-go/aws"
   "github.com/aws/aws-sdk-go/aws/session"
   "github.com/aws/aws-sdk-go/service/elasticache"
   "time"
)

func main() {

   endpoints := []string{}
   client := elasticache.New(session.New(),&aws.Config{Region: aws.String("eu-west-1")})
   params := &elasticache.DescribeCacheClustersInput{
      CacheClusterId:    aws.String("eticket-cache"),
      MaxRecords:        aws.Int64(20),
      ShowCacheNodeInfo: aws.Bool(true),
   }

   go func(endpoints *[]string) {
      for {
         *endpoints = nil
         resp, err := client.DescribeCacheClusters(params)

         if err == nil {
            for _, cluster := range resp.CacheClusters {
               for size, node := range cluster.CacheNodes {
                  if size == 0 {
                     if *cluster.CacheNodes[0].CacheNodeStatus == "available" {
                        *endpoints = append(*endpoints, fmt.Sprintf("%s:%d", *cluster.CacheNodes[0].Endpoint.Address, *cluster.CacheNodes[0].Endpoint.Port))
                     }
                  } else {
                     if *node.CacheNodeStatus == "available" {
                        *endpoints = append(*endpoints, fmt.Sprintf("%s:%d", *node.Endpoint.Address, *node.Endpoint.Port))
                     }
                  }
               }
            }
            time.Sleep(time.Second * 2)
         }
      }
   }(&endpoints)

   for {
      time.Sleep(time.Second * 2)
      if endpoints != nil {
         fmt.Println(endpoints);
      }
   }

}

You only need to replace the region and CacheClusterId with your own. You can find the CacheClusterId in your ElastiCache console:

cluster

DescribeCacheCluster returns a lot of information regarding your cache cluster:

{
  CacheClusters: [{
      AutoMinorVersionUpgrade: true,
      CacheClusterCreateTime: 2016-06-20 09:39:44.379 +0000 UTC,
      CacheClusterId: "eticket-cache",
      CacheClusterStatus: "available",
      CacheNodeType: "cache.t2.micro",
      CacheNodes: [{
          CacheNodeCreateTime: 2016-06-20 09:39:44.379 +0000 UTC,
          CacheNodeId: "0001",
          CacheNodeStatus: "available",
          CustomerAvailabilityZone: "eu-west-1a",
          Endpoint: {
            Address: "eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com",
            Port: 11211
          },
          ParameterGroupStatus: "in-sync"
        },{
          CacheNodeCreateTime: 2016-07-12 13:57:48.598 +0000 UTC,
          CacheNodeId: "0002",
          CacheNodeStatus: "available",
          CustomerAvailabilityZone: "eu-west-1a",
          Endpoint: {
            Address: "eticket-cache.4xznmr.0002.euw1.cache.amazonaws.com",
            Port: 11211
          },
          ParameterGroupStatus: "in-sync"
        }],
      CacheParameterGroup: {
        CacheParameterGroupName: "default.memcached1.4",
        ParameterApplyStatus: "in-sync"
      },
      CacheSubnetGroupName: "cache-subnet",
      ClientDownloadLandingPage: "https://console.aws.amazon.com/elasticache/home#client-download:",
      ConfigurationEndpoint: {
        Address: "eticket-cache.4xznmr.cfg.euw1.cache.amazonaws.com",
        Port: 11211
      },
      Engine: "memcached",
      EngineVersion: "1.4.24",
      NumCacheNodes: 2,
      PendingModifiedValues: {

      },
      PreferredAvailabilityZone: "eu-west-1a",
      PreferredMaintenanceWindow: "sun:23:00-mon:00:00",
      SecurityGroups: [{
          SecurityGroupId: "sg-7d91121a",
          Status: "active"
        }]
    }]
}

We only need to know the endpoint address and port.

Running the demo application

When running the demo application remember that your EC2 instance must have AWS credentials (environment variables, shared credentials file or IAM roles).

You’ll get an output similar to this if you have one node:

[eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211]

If you add a new node the application will automatically identify a new node is available and it will add it to the slice:

[eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211 eticket-cache.4xznmr.0002.euw1.cache.amazonaws.com:11211]

Conclusion

Using node Auto Discovery is very useful since it avoids any manual intervention. Just add or remove a new node and your GO application will identify the changes automatically.