Give Mesos and External Volumes a spin with playa-mesos

OLD - {code} by Dell EMC

Mesos is an important platform to consider if you’re interested in running containers in a highly available manner, operating an Enterprise-friendly container platform, or building application platforms to operate complex distributed applications. It should be thought of in a collaborative and complimentary way to the container eco-system. For some, it will sit at the scheduling layer only, and for others it will span across scheduling to the container runtime. Mesos represents a new way of thinking when it comes to how we operate and consume data center resources.

The Mesos platform is often adopted when a data center is moving towards the following key points:

  • A homogenous operating environment where all compute resources can run all workloads
    • Data center silos for workloads that aren’t virtualization friendly can now be scheduled alongside other workloads (ie. Hadoop and Cassandra)
    • IaaS and virtualization are no longer needed to pool resources
  • Providing simple but highly available applications
    • Basic capabilities include…

View original post 753 more words

Is Neural Network Better Off with Big Data


How does neural network or for that matter any machine learning model relates to Big Data. Do we get a better quality learning model with bigger data. That’s what we will explore in this post. We will explore sample complexity i.e. the way model performance varies with training sample size. This will be particularly interesting from a Big Data point of view.  We will also look at model complexity which tells us how model performance varies with model complexity.

Although I have used a multi layer neural network for my experiments, the findings should

View original post 1,696 more words

Raspberry Pi Lights: how to sync Christmas lights to midi audio

The Raspberry Pi and I

** UPDATE 9/21/2014 **

I updated the source code today. Now lightorgan supports more than 7 output channels.  It chooses the pin to light up based on both the pitch and the octave of every note. Now the number of supported output channels is limited only by how dynamic the range of the midi file is. I observed that this worked practically for at least 24 channels on several Christmas songs. This is cool because the new Rasperry Pi Model B now supports up to 28 pins!  See Gordon’s page at

Also, the WiringPi pins that lightorgan uses are now configurable. Just modify the array called pinMapping[] that’s near the top of the lightorgan.c file to add, remove, or remap a lightorgan channel to a corresponding WiringPi  pin. Recompile with your changes and then you should be good to go.

Check out the new source code from the google…

View original post 1,287 more words

Backup & Restore Your Logstash/Grafana Dashboards

Web Development Insights

I created a Chef cookbook with which you can backup and restore your Logstash/Grafana dashboards. You can find it here.

It wouldn’t be an exaggeration to say that Logstash and Grafana have changed my life. I can’t even remember how I was monitoring or investigating performance issues before having them. When I first installed those tools and started to feed them with data I was really excited by the possibilities they offered me. Building dashboards was so easy and fluid. Create a widget, select the data to display and viola – you have a neat looking graph! With time I added more and more dashboards both to Logstash and Grafana. I now have dozens in each. They show me everything I need to know and when I find they have a missing piece – I add it right away.

A sample Grafana dashboard A sample Grafana dashboard

Everything went well, life…

View original post 878 more words

Why We Chose Kubernetes Over ECS

Web Development Insights

On our last post, we saw how Docker changed the way we treat our infrastructure and what changes it brought to the domain of service orchestration.
In the following post, we’re going to take a tour of two of the leading Docker orchestration frameworks out there: ECS (Elastic Container Service) by AWS, and Kubernetes, an orchestration framework which began at Google and became open source later.

3 months ago when we, at, came to evaluate which Docker orchestration framework to use, we gave ECS the first priority. We were already familiar with AWS services, and since we already had our whole infrastructure there, it was the default choice. After testing the service for a while we had the feeling it was not mature enough and missing some key features we needed (more on that later), so we went to test another orchestration framework: Kubernetes. We were glad to discover that…

View original post 2,447 more words

Apache Spark Cluster Internals: How spark jobs will be computed by the spark cluster

Knoldus Blogs

In this blog we are explain how the spark cluster compute the jobs. Spark jobs are collection of stages and stages are collection of tasks. So before the deep dive first we see the spark cluster architecture.


In the above cluster we can see the driver program it is a main program of our spark program, driver program is running on the master node of the spark cluster.

Cluster manager is the responsible for allocating resources for the given job.

And worker nodes have a executers in which the task will be running and stored the data in the cache.

This is Apache Spark basic architecture of the cluster.

Now we discuss about different RDD types created on transformations as follows:

  • HadoopRDD
  • FilteredRDD
  • ShuffleRDD

HadoopRDD: Spark make a RDD from the Hadoop InputFormat so it makes a new HadoopRDD and map the partitions with Hadoop block size by default…

View original post 325 more words

Using a file as a device in ScaleIO

Kalle's playground

In most cases one would use the whole unformatted disk as a device for ScaleIO, but sometimes this is not possible. To use the free space from already partitioned disk ScaleIO sds component has a command line tool called create_file_storage. It is located in the ScaleIO sds component folder. Tool will create a pre-allocated file with specified size to a specified location.

Command to create file:

create_file_storage −−create_file −−size_gb <SIZE_IN_GB> −−file_name <FILE_NAME> 


create_file_storage −−create_file −−size_gb 120 −−file_name

create_file_storage command line

When adding a devices for SDS specify the full path of the file as the “Path” for the device.

File as device

The “file as device” options allows to start consuming free space for ScaleIO from disks that have existing partitions and existing file systems.

View original post

Run your stateful apps with Mesos and Docker

OLD - {code} by Dell EMC

Hot off the heels of ContainerCon/MesosCon we looked into how we could help support data persistence for the popular distributed systems manager. We trawled through a lot of documentation, and found that Marathon has a great Docker containerizer and also supports arbitrary options. Yay! Have we lost you yet?

Here’s how it works:

Mesos is a tool to abstract datacenter resources such as CPU, memory, storage and more. Marathon is used on top of Mesos to schedule applications on top of those resources, and those applications can be in Docker containers. As you might remember from previous posts, the EMC {code} team have worked on a Docker volume plugin called REX-Ray (new name suggestions welcome) to make sure that we can actually store data from our containerized applications. So, with all these pieces together, we should be able to create a volume using REX-ray, tie it to a container, have…

View original post 672 more words

There’s Real Magic behind OpenStack Neutron

Tricky Deadline

Really easy to create a new post on any blog – the most complex part is to think about the content – you can do it through a couple of clicks on your laptop, or from your phone/tablet, or just reblog other interesting posts from some mates. The same thing happens with OpenStack, you can create Servers instances, routers, subnets, firewalls, VPNs, Load Balancers through just a couple of clicks – well, first of all,  you need to plan and figure it why you need to create them, it’s not just create things because you can, depending on the application this virtual infrastructure will support, you need to define what your virtual IT architecture will be –

In my previous note about Neutron I’ve mentioned how Neutron brings Layer 3 capabilities and how easy is to create and manage it. Now I will describe the magic behind, I’m sure you will…

View original post 808 more words