AWS SDK ELASTICACHE AUTO DISCOVERY FOR GO

In this tutorial I’ll show you how to implement your own ElastiCache (Memcached) node Auto Discovery. Auto Discovery is useful when you need to add or remove nodes without having to manually add/remove the endpoints in your application. If in your application you’re only using one node your code would look like this:

mc := memcache.New("eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211")

If that memcached node began to evict items a new node would be needed. Adding a new node in AWS easy, go to your Cache Cluster console and click on the “Add Node” button. In a couple of minutes your node will up & running.

nodes

Since your application does not have any node Auto Discovery code you would need to add that new node manually:

mc := memcache.New("eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211", "eticket-cache.4xznmr.0002.euw1.cache.amazonaws.com:11211")

Then you’d build & upload your project to your server or servers. All these steps can be avoided by implementing node Auto Discovery.

The Code

package main

import (
   "fmt"
   "github.com/aws/aws-sdk-go/aws"
   "github.com/aws/aws-sdk-go/aws/session"
   "github.com/aws/aws-sdk-go/service/elasticache"
   "time"
)

func main() {

   endpoints := []string{}
   client := elasticache.New(session.New(),&aws.Config{Region: aws.String("eu-west-1")})
   params := &elasticache.DescribeCacheClustersInput{
      CacheClusterId:    aws.String("eticket-cache"),
      MaxRecords:        aws.Int64(20),
      ShowCacheNodeInfo: aws.Bool(true),
   }

   go func(endpoints *[]string) {
      for {
         *endpoints = nil
         resp, err := client.DescribeCacheClusters(params)

         if err == nil {
            for _, cluster := range resp.CacheClusters {
               for size, node := range cluster.CacheNodes {
                  if size == 0 {
                     if *cluster.CacheNodes[0].CacheNodeStatus == "available" {
                        *endpoints = append(*endpoints, fmt.Sprintf("%s:%d", *cluster.CacheNodes[0].Endpoint.Address, *cluster.CacheNodes[0].Endpoint.Port))
                     }
                  } else {
                     if *node.CacheNodeStatus == "available" {
                        *endpoints = append(*endpoints, fmt.Sprintf("%s:%d", *node.Endpoint.Address, *node.Endpoint.Port))
                     }
                  }
               }
            }
            time.Sleep(time.Second * 2)
         }
      }
   }(&endpoints)

   for {
      time.Sleep(time.Second * 2)
      if endpoints != nil {
         fmt.Println(endpoints);
      }
   }

}

You only need to replace the region and CacheClusterId with your own. You can find the CacheClusterId in your ElastiCache console:

cluster

DescribeCacheCluster returns a lot of information regarding your cache cluster:

{
  CacheClusters: [{
      AutoMinorVersionUpgrade: true,
      CacheClusterCreateTime: 2016-06-20 09:39:44.379 +0000 UTC,
      CacheClusterId: "eticket-cache",
      CacheClusterStatus: "available",
      CacheNodeType: "cache.t2.micro",
      CacheNodes: [{
          CacheNodeCreateTime: 2016-06-20 09:39:44.379 +0000 UTC,
          CacheNodeId: "0001",
          CacheNodeStatus: "available",
          CustomerAvailabilityZone: "eu-west-1a",
          Endpoint: {
            Address: "eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com",
            Port: 11211
          },
          ParameterGroupStatus: "in-sync"
        },{
          CacheNodeCreateTime: 2016-07-12 13:57:48.598 +0000 UTC,
          CacheNodeId: "0002",
          CacheNodeStatus: "available",
          CustomerAvailabilityZone: "eu-west-1a",
          Endpoint: {
            Address: "eticket-cache.4xznmr.0002.euw1.cache.amazonaws.com",
            Port: 11211
          },
          ParameterGroupStatus: "in-sync"
        }],
      CacheParameterGroup: {
        CacheParameterGroupName: "default.memcached1.4",
        ParameterApplyStatus: "in-sync"
      },
      CacheSubnetGroupName: "cache-subnet",
      ClientDownloadLandingPage: "https://console.aws.amazon.com/elasticache/home#client-download:",
      ConfigurationEndpoint: {
        Address: "eticket-cache.4xznmr.cfg.euw1.cache.amazonaws.com",
        Port: 11211
      },
      Engine: "memcached",
      EngineVersion: "1.4.24",
      NumCacheNodes: 2,
      PendingModifiedValues: {

      },
      PreferredAvailabilityZone: "eu-west-1a",
      PreferredMaintenanceWindow: "sun:23:00-mon:00:00",
      SecurityGroups: [{
          SecurityGroupId: "sg-7d91121a",
          Status: "active"
        }]
    }]
}

We only need to know the endpoint address and port.

Running the demo application

When running the demo application remember that your EC2 instance must have AWS credentials (environment variables, shared credentials file or IAM roles).

You’ll get an output similar to this if you have one node:

[eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211]

If you add a new node the application will automatically identify a new node is available and it will add it to the slice:

[eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211 eticket-cache.4xznmr.0002.euw1.cache.amazonaws.com:11211]

Conclusion

Using node Auto Discovery is very useful since it avoids any manual intervention. Just add or remove a new node and your GO application will identify the changes automatically.

AWS ElastiCache Memcached cluster in a GO project

In this tutorial we’ll learn how to use AWS ElastiCache Memcached cluster in a GO project.
We’ll be using the most recommended GO library: https://github.com/bradfitz/gomemcache

To follow this tutorial you’ll need a working ElastiCache Memcached cluster. Read the AWS getting started tutorial.

Configuring

go get github.com/bradfitz/gomemcache/memcache

Install git if you get the following error:

go: missing Git command. See https://golang.org/s/gogetcmd
package github.com/bradfitz/gomemcache/memcache: exec: "git": executable file not found in $PATH

The code

package main

import (
    "fmt"
    "github.com/bradfitz/gomemcache/memcache"
    "log"
    "os"
    "errors"
)

func main() {

    if len(os.Args) < 3 {
        log.Fatalln(errors.New("Usage: main key value"))
    }

    key := os.Args[1]
    value := os.Args[2]

    mc := memcache.New("eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211")
    mc.Set(&memcache.Item{Key: key, Value: []byte(value)})

    it, err := mc.Get(key)

    if err != nil {
        fmt.Println(err)
    } else {
        fmt.Println(it)
        fmt.Println(string(it.Value[:]))
    }
}

You must replace the Memcached endpoints with your own. Go to your ElastiCache Dashboard, Cache Clusters & click on Nodes.
Click on “Copy Node Endpoint” to view all your available Memcached node endpoints.

memcached4

If you have more than one node edit the memcache.New line:

mc := memcache.New("eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com:11211", "eticket-cache.4xznmr.0002.euw1.cache.amazonaws.com:11211")

Build and run the application

go build main.go
./main Arg1 Arg2

Replace Arg1 & Arg2 with the key and value you wish to create.

Testing

You can test that your keys are being created by clicking on the node(s) and scroll to the “Current Items (Count)” graph.

memcached3

From your EC2 instance you can telnet to your Memcached node and run commands to query the keys stored in that server.

· stats items retrieves stats from the node
· stats cachedump 1 5 retrieves keys 1 to 5

telnet eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com 11211
[root@ip-10-0-1-39 ec2-user]# telnet eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com 11211
Trying 10.0.1.20...
Connected to eticket-cache.4xznmr.0001.euw1.cache.amazonaws.com.
Escape character is '^]'.
stats items

STAT items:1:number 1
STAT items:1:age 276
STAT items:1:evicted 0
STAT items:1:evicted_nonzero 0
STAT items:1:evicted_time 0
STAT items:1:outofmemory 0
STAT items:1:tailrepairs 0
STAT items:1:reclaimed 0
STAT items:1:expired_unfetched 0
STAT items:1:evicted_unfetched 0
STAT items:1:crawler_reclaimed 0
STAT items:1:crawler_items_checked 0
STAT items:1:lrutail_reflocked 0
END
stats cachedump 1 5

ITEM foo [8 b; 1466415492 s]
END

Conclusion

In this simple example we connected to a Memcached server, created a key & retrieved its value. In the next tutorial we’ll learn how to add Auto Discovery to add/remove nodes dynamically.

GO AWS SDK

In this tutorial I’ll show you how to configure GO in an AWS EC2 instance and use the AWS GO SDK.

Configuring GO

Connect to your EC2 instance

ssh ec2-user@PUBLIC_IP -i yourpemfile.pem

Execute the following commands to install & configure GO

sudo su
mkdir /downloads
cd downloads
wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz
tar xvfz go1.6.2.linux-amd64.tar.gz -C /usr/local
cd /usr/local/go
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin
cd $HOME
mkdir work
export GOPATH=$HOME/work
cd work
mkdir bin | mkdir pkg | mkdir src

You should put the export commands in your shell startup script ($HOME/.bashrc)

vi $HOME/.bashrc

export GOPATH=$HOME/work
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin

You now have GO configured in your EC2 instance. You can test that it’s working by executing the following commands

cd src | mkdir aws-sdk-go
cd aws-sdk-go

vi main.go
	
package main

import "fmt"

func main() {
	 fmt.Printf("hello, world\n")
}

go run main.go

You should see the text “hello, world” when running the go command.

AWS credentials

To access AWS services you need to set your account credentials. You have three valid methods:

  1. Environment variables

    export AWS_ACCESS_KEY_ID=YOUR_AKID
    export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
  2. Shared credential files

    Edit ~/.aws/credentials
    
    [default]
    aws_access_key_id = ACCESS_KEY_ID
    aws_secret_access_key = SECRET_ACCESS_KEY_ID
  3. EC2 instance profile credentials (IAM roles) This is the preferred method if running on EC2
    A) Create an IAM Role S3 Access
    B) Assign the role to the EC2 instance

You should never hardcode your credentials in your code. It makes it difficult to maintain (if keys are changed for example) and they might be exposed by mistake.

Let’s download the AWS SDK and create our S3 test app.

go get -u github.com/aws/aws-sdk-go/...

Edit the main.go file:

package main

import (
        "fmt"
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/service/s3"
        "strings"
)

func main() {
        client := s3.New(session.New(),&aws.Config{Region: aws.String("eu-west-1")})
        result, err := client.PutObject(&s3.PutObjectInput{
                Bucket: aws.String("yourbucketname"),
                Key:    aws.String("testkey"),
                Body:   strings.NewReader("Hello"),
        })
        if err != nil {
                fmt.Println(err)
        }else{
                fmt.Println(result)
        }
}

You should change yourbucketname with a bucket that exists in your AWS account. You should also set the region bucket, changing eu-west-1 with your desired region.

Time to test your first GO AWS SDK app:

go run main.go

You should get a response similar to this:

[root@ip-10-0-1-39 ec2-user]# ./main 
{
  ETag: "\"8b1a9953c4611296a827abf8c47804d7\""
}

If your credentials are incorrect you will get a response similar to this:

[root@ip-10-0-1-39 ec2-user]# ./main 
InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: C2A2BA80682D8ABB

Conclusion

In this short tutorial you’ve learned how to configure GO to access AWS services using Amazon’s SDK for GO. If you want to learn more about the AWS SDK for GO: http://docs.aws.amazon.com/sdk-for-go/latest/v1/developerguide/sdkforgo-dg.pdf