Data Pipeline Service Graph Using AWS Neptune Part 1

Background

todo

Goals

todo

The Journeys

todo

todo

We will deploy a Neptune instance on AWS using Cloudformation below.

We also need to have a file called neptune.params.prod.json, which define all parameters required by Cloudformation template. Below is one example.

To deploy the stack, let’s use below command:

aws cloudformation create-stack \
--stack-name neptune-data-pipeline-graph-prod \
--template-body file://neptune-cfn.yaml \
--parameters file://neptune.params.prod.json

Let us summarize what above Cloudformation YAML and deploy command does.

  • The stack will use existing VPC Id, Subnet IDs (at least 2), and SNS Topic (ARN). So please make sure you created one and mention it on neptune.params.prod.json .
  • The stack will create below resources:
Resource Type                         Resource Logical IdAWS::CloudWatch::Alarm                NeptunePrimaryCpuAlarm
AWS::CloudWatch::Alarm NeptunePrimaryGremlinRequestsPerSecAlarm
AWS::CloudWatch::Alarm NeptunePrimaryMemoryAlarm
AWS::CloudWatch::Alarm NeptunePrimarySparqlRequestsPerSecAlarm
AWS::EC2::SecurityGroup NeptuneDBSG
AWS::IAM::ManagedPolicy NeptuneCloudWatchPolicy
AWS::IAM::ManagedPolicy NeptuneS3Policy
AWS::IAM::Role NeptuneRole
AWS::Neptune::DBCluster NeptuneDBCluster
AWS::Neptune::DBClusterParameterGroup NeptuneDBClusterParameterGroup
AWS::Neptune::DBInstance NeptuneDBInstance
AWS::Neptune::DBParameterGroup NeptuneDBParameterGroup
AWS::Neptune::DBSubnetGroup NeptuneDBSubnetGroup

Neptune database will be there in AWS console.

We will use a very cool tools called Graphexp to explore and visualize (even do some editing) Neptune database we created earlier.

Download release ZIP file from https://github.com/bricaud/graphexp/releases. For example I will use latest version at the time writing this article which is 0.8.0.

# Download source code
wget https://github.com/bricaud/graphexp/archive/v0.8.0.zip
# Extract the zip
unzip v0.8.0.zip
# Enter the directory
cd graphexp-0.8.0/

Edit file scripts/graphioGremlin.js . Find function called send_to_server , replace http to https and ws to wss .

Serve using simple Python server:

python -m SimpleHTTPServer 5000

Graphexp will run locally on http://localhost:5000.

Now, we need to create a tunnel between our local machine to Neptune instance via an EC2 bastion host instance as described here.

ssh -L 8182:mydbcluster.cluster-123456789012.eu-west-1.neptune.amazonaws.com:8182 -i my_keypair.pem ec2-user@ec2-11-111-11-111.eu-west-1.compute.amazonaws.com

Now, test on http://localhost:5000

We’re ready to do some interesting things on Part 2.

--

--

--

Data & Telco

Love podcasts or audiobooks? Learn on the go with our new app.

What Does a Standard GL Integration Using Salesforce Look Like?

Deploying Machine learning Models With Docker

Stimulsoft Reports and Dashboards. New version 2020.1 released

[LeetCode] #1551. Minimum Operations to Make Array Equal

Development

How to become a web designer on budget -Day11

Working of the TCP/IP Layers during a Network Flow

10 Up & Comers to watch in the Enterprise Web Development Industry

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Taufiq Ibrahim

Taufiq Ibrahim

Data & Telco

More from Medium

How We Provide a Large Number of Cloud Services and Clusters and Live to Tell the Tale

Transactional Replication b/w SQL Server and AWS RDS

Migrating Transactional Data to MongoDB in AWS with the Zaloni Arena Data Governance Platform

AWS access keys IP filtering protection