Data Pipeline Service Graph Using AWS Neptune Part 1

Background

todo

Goals

todo

The Journeys

todo

1 Creating Solution Architecture

todo

2 Deploying AWS Neptune DB

We will deploy a Neptune instance on AWS using Cloudformation below.

We also need to have a file called neptune.params.prod.json, which define all parameters required by Cloudformation template. Below is one example.

To deploy the stack, let’s use below command:

aws cloudformation create-stack \
--stack-name neptune-data-pipeline-graph-prod \
--template-body file://neptune-cfn.yaml \
--parameters file://neptune.params.prod.json

Let us summarize what above Cloudformation YAML and deploy command does.

• The stack will use existing VPC Id, Subnet IDs (at least 2), and SNS Topic (ARN). So please make sure you created one and mention it on neptune.params.prod.json .
• The stack will create below resources:

Resource Type                         Resource Logical Id
AWS::CloudWatch::Alarm                NeptunePrimaryCpuAlarm
AWS::CloudWatch::Alarm     NeptunePrimaryGremlinRequestsPerSecAlarm
AWS::CloudWatch::Alarm                NeptunePrimaryMemoryAlarm
AWS::CloudWatch::Alarm     NeptunePrimarySparqlRequestsPerSecAlarm
AWS::EC2::SecurityGroup               NeptuneDBSG
AWS::IAM::ManagedPolicy               NeptuneCloudWatchPolicy
AWS::IAM::ManagedPolicy               NeptuneS3Policy
AWS::IAM::Role                        NeptuneRole
AWS::Neptune::DBCluster               NeptuneDBCluster
AWS::Neptune::DBClusterParameterGroup NeptuneDBClusterParameterGroup
AWS::Neptune::DBInstance              NeptuneDBInstance
AWS::Neptune::DBParameterGroup        NeptuneDBParameterGroup
AWS::Neptune::DBSubnetGroup           NeptuneDBSubnetGroup

Neptune database will be there in AWS console.

3 Using Graphexp To Explore Neptune Graph

We will use a very cool tools called Graphexp to explore and visualize (even do some editing) Neptune database we created earlier.

Download release ZIP file from https://github.com/bricaud/graphexp/releases. For example I will use latest version at the time writing this article which is 0.8.0.

# Download source code
wget https://github.com/bricaud/graphexp/archive/v0.8.0.zip
# Extract the zip
unzip v0.8.0.zip
# Enter the directory
cd graphexp-0.8.0/

Edit file scripts/graphioGremlin.js . Find function called send_to_server , replace http to https and ws to wss .

Serve using simple Python server:

python -m SimpleHTTPServer 5000

Graphexp will run locally on http://localhost:5000.

Now, we need to create a tunnel between our local machine to Neptune instance via an EC2 bastion host instance as described here.

ssh -L 8182:mydbcluster.cluster-123456789012.eu-west-1.neptune.amazonaws.com:8182 -i my_keypair.pem ec2-user@ec2-11-111-11-111.eu-west-1.compute.amazonaws.com

Now, test on http://localhost:5000

We’re ready to do some interesting things on Part 2.