Docker on Mac OS 10.10.

Well, Docker is everywhere now!! For last few weeks studying their applicability in DevOps, Micro services etc. Today wanted to make my hands dirty with some Docker samples. But its not smooth start. Just wanted to share the issues and resolution at one place.

After installing docker as per documentation on Mac (OS X 10.10) and launching the CLI first error got was “Default is not running. Please start this with docker-machine start default.” After googling around and analysing different solutions, found the issue is with Virtualbox version. (I had Viirtualbox version 5.0 already installed.). The issue is resolved with installing test Test-Build 5.0 (https://www.virtualbox.org/wiki/Testbuilds). The link for issue is  https://www.virtualbox.org/ticket/14412 

After restarting Docker, next error I got is with ca.pem file. After starting Default VM error shown is “ca.pem: no such file or directory”. After executing below two commands, docker started normally

docker-machine regenerate-certs default

docker-machine restart default

Below are the links that helped me solve the issues

https://github.com/docker/docker/issues/15712

http://stackoverflow.com/questions/32240742/after-installing-docker-docker-machine-env-default-fails-because-ca-pem-is-not

Code on Your Own device (CYOD)

In my blog post in Feb, 2011  "Work on move... or coding on mobile" I envisioned possibility of coding on a mobile device. Today I was talking to my colleague who said he uses only iPad for work no Laptop!!. I asked how do you code? He connects to server using Microsoft Remote Desktop, connects a key board to iPad and codes!!. Some time back from I got to know  from IoT team of a light weight version of eclipse that works on Raspberry PI which got 1GB RAM. Today I use a phone that got 3GB RAM!!!. Don't think its long way to have Eclipse and JRE on Mobile devices. (I guess there are some issues to install eclipse directly now but there are alternatives like Ubuntu Virtual Box on Android and Eclipse on top of it, root the device and install Linux etc).

So can we see "Code on Your Own device" movement soon!!!

Facebook for Business - Mind Map

I tried to summarize my understanding of Facebook's applicability for Business. 

Apart from Facebook's Graph API and Ads API below are few things worth learning about

•Open Graph

–Open Graph lets apps tell stories on Facebook through a structured, strongly typed API.

–Stories used share the things people are doing on webpage/App. Ex: “X has finished reading Y book on Google Reads”, “A bought product B on XYZ Store”. These are posted on user’s Facebook page as posts

–Insights API can be used to track the stories, likes and comments on them.

•Facebook exchange

–Facebook Exchange helps you reach people who have expressed an interest through their online behavior so you can reach them with a similar kind of product or service.

–Ex: Target user who searched for product or clicked a web site.

–Uses Demand-side platforms

•PMD Center

–The Facebook Preferred Marketing Developer (PMD) program is a community of hundreds of best-in-class developers /companies Facebook marketing solutions.

–Companies get additional support from Facebook platform representatives

•Public Content Solutions (PCS)

–Partnership with companies which  are experts in integrating Facebook and Instagram into their customers’ experiences connecting audiences and fans

–Privileged access to data and content APIs and dedicated support resources

–Direct collaboration with Facebook business development to help support and grow your business

Issue with Facebook Graph API

Facing an issue with Facebook API. Looking out for help to understand the reasons and resolve the same
We are trying get public wall messages containing a particular keyword.We created application with permissions including read_stream and manage_posts. Application visibility is public and most of other settings are defaults. We could see API working and getting the feeds. Last week we observed an issue. Search API is not picking up messages from certain users. We picked up couple of such ids did some testing. Privacy settings all seems to be fine. Posts visibility settings are public both at feed level and account leven. Platform is turned on. Visibility to search engines is on. For these selected IDs we tried to get wall feeds using their IDs. Below are the observations.

1. For a certain ID we could get all messages both in search and wall retrieval. Both with our application key and default Graph API Explorer access key.
2. For one ID we for messages using Graph API Explorer but not with our application. With our application only feeds we see response are friend request accepts and likes to some other public posts.
3. For another ID Graph API explorer is giving empty results and out application key is giving likes etc as observed in above case.
4. For one of the Use we asked him to turn off the Platform, we were getting messages before. When turned off we stopped getting messages. When turned on we didn't get all messages but only likes etc.
5. When we use Social Mention (socialmention.com) and search for keyword, we could see all messages.

We feel it has to do something with the Application and access key permissions but couldn't get much details. Any help in cracking this is welcome.

Update (May 05, 2014):
The issue is with the way access token generated.

While generating access key we can select permissions to match application requirements. Any thing beyond basic permissions require permission from User. For example if we choose user_location permission, user need to provide permission to get that information. If no additional permissions selected, application will not require any permission and can access all the public data. But if any additional permissions are selected and user not given permission, we cannot get even the public data of that user. With default permissions I generated access key and could get all public data as expected.

"Apache Flume: Distributed Log Collection for Hadoop" - Book review

We have been working on Apache Flume for quite some time now. We used it to load data from Social network into MongoDB and also for log collection. Recently I read a book on Apache Flume titled "Apache Flume: Distributed Log Collection for Hadoop".

 

This is a good starter material for a serious Flume developer. 2nd Chapter provides a good step by step guide to setup and get running with Flume. Liked the way the flow is presented and the mention about important arguments that can be provided to Flume command line. Other useful information from this book is about the Monitoring tools. Though the description is brief good to have introduction to some tools. Overall book gives good details with examples on Flume flow and architecture which includes Channels, agents, sinks, interceptors etc. It would have been more helpful if last two chapters were elaborated more.

Modes of Big Data Analysis

We can look at analysis in three modes based on trigger for analysis.

• Offline/Batch Mode
• Analytics performed and results are made available for applications to use
• Ex: Clinical Trails, Voice of Customer
• Real Time – OnDemand
• Analysis done and results are presented when requested.
• Ex: Up-sell/Cross-sell
• Real Time – Stream based
• Monitor streaming data (Twitter messages, Transaction logs, data from Sensors) and trigger analysis based on event/data.
• Ex: Monitor and analyze online transactions for Fraud, Monitor social media messages for serious incidents.

 



And below are the implementation approaches:

• Massive Parallel Programming (Data Bases and Programming)Hadoop MapReduce
• Scalable Database – NoSQL databases and Databases with ability to store huge data (Ex Oracle ExaData) and to perform operations on data.
• In-memory Analytics - an approach to querying data when it resides in random access memory (RAM), as opposed to querying data that is stored on physical disks.
• Big Data Appliance - combination hardware and software products designed specifically for analytical processing.
• Processing in Memory (PIM) - a chip architecture in which the processor is integrated into a memory chip to reduce latency.
• In-Database Analytics - a technology that allows data processing to be conducted within the database by building analytic logic into the database itself.
• Real-time Stream Processing & CEP

Combination of above approaches need to implement the Analytic Apps

 

Almost 2 years back, for couple of months, I had my first stint with Big data and Hadoop before moving on to Social Analytics. As I resumed my interest into Big data I was looking at my old work and above are from one of my early presentations.

Klout4Java : Java wrapper for Klout partner API

One more contribution for Open source. Uploaded klout4java, a Java wrapper for Klout Partner API v2. I had calls to klout API in our Social Analytic platform. As v1 of klout API getting deprecated, I had to upgrade and I couldn't find any Java wrapper for v2. This lead me to write a wrapper for myself and I uploaded the same for community use.

The link to klout4java http://code.google.com/p/klout4java/