Satyameva Jayate - Twitter (Sentiment) Analysis

As we all know Social media is changing or influencing lot of the traditional communication mechanisms. Its providing more wide platform for people to share thoughts, advertise products and analyze user's feelings. For those close to technology, Social Media monitoring, Social Analytics, Social Engineering etc are buzzwords heard more often these days. Its hard for any body to ignore Social Media now. From my experience so far FMCG, Entertainment media are among the areas that has their presence more in Social Media. Its very common to see references to Twitter in print and electronic media these days.

Amir Khan's completed his Satyameva Jayate's (SMJ) first season. keeping aside the sensitiveness of topics, accuracy of facts, its impact on society etc, SMJ is certainly one of the shows which is talked about more in Social Media. Thought of using my experience in Social Analytics to present some of the analysis on SMJ from twitter feeds. I will start with basic analysis in this post.

First lets see the tweet density, the number of related tweets during the 13 week time. The peaks that you observe are, as expected, are on the days the show telecast.Overall trend line show little decline in tweet. Also few episodes in between are less popular. These are when topics like domestic voilance, use fertilizers etc discussed. Alcohol Abuse, Aging parents, Water conservation topics got some increased attention on twitter. Of course, this might reflect only immediate reaction to show. Will try to present detailed analysis by presenting average week analysis and topic wise analysis in future posts. Next lets see the Sentiment Analysis.

Lets first look at the Positive trend, that is the percentage of the total tweets with positive sentiment over period of time. Adjacent graph depicts the positive sentiment. Overall trend line shows a little decline in positive sentiment. From initial observation it looks tweets during episode on medicine bear more positive sentiment compared to other episodes. Again, this graph reflects overall mood of the user in general. More deeper analysis, like feature based Sentiment Analysis, is required to capture exact mood of the audience. It will answer questions like, is this positive sentiment during healthcare episode is towards Amir's views or towards Health care industry? 

Finally lets look at Negative trend graph. Graph on left hand side shows the same. As observed the overall trend line shows increase in Negative trend. As mentioned earlier will provide deeper analysis in future posts. Main intention of this post is not to do postmortem of SMJ but to show some applications of Social Analytics with an interesting and relevant case study.

Before closing, a word about data collection and analysis techniques. Data was collected with Twitter Search API using keywords. Of the keywords used SMJ, Satyameva Jayate, Amir Khan, SatyamevJayate, smjindia are the common keywords used for all episodes and episode specific keywords like AlcoholAbuse, dignity4all etc based are used. I wont rule out possibility of missing some tweets partly because of timing of search and partly because of accuracy of Search API itself. I generally executed Search program couple of times on Sunday and once during week. For Sentiment Analysis used Sentistrength Algorithm (http://sentistrength.wlv.ac.uk/) with modified Lexicon. Again the accuracy of this Sentiment Analysis might not be high. I expect this to be around 65%. In fact from our experience so far this is reasonable accuracy especially for Social Media data which doesn't follow language grammar. For more information on Sentiment analysis of Twitter feeds look at this white paper I co-authored - http://t.co/aXAv7aly.

JSF Validations, Dependency injection etc

For last few months I am getting to customize quite a few JSF features. This is JSF 1.1 and I some times I feel some level of dependency injection would have made my life easy. I shall disuss couple of examples below.
As in struts validations kicks in whenever you submit a form. But there are many instances where I need to submit form (bcause I need all values in screen) but don't perform validatiions for one action on a screen and for other action I need to perform validations. Basically I like to skip validations if there is a reauest parameter named SkipValidation present or target method name starts with 'load'.
In struts I achieved this by overriding processValidation method and adding a condition so that validation is skipped if method name follows certain pattern. I thought I can do a simillar thing in jsf. Wanted to modify ValidationPhase such that validation is skipped in certain cases.
But the class that implements validations is hardcoded or instanciated in LifecycleImpl class. To create a custom life cycle, you need to create Impl class configure it web.xml, create ContextListener to populate FacesConfig and bind all faces listeners to new impl class. And I add no to minimal code in tbhese classes. If I could inject the class that implements validation phase using some config file via dependency injection, I could have avoided lot of coding.
I will discuss another instancd onvolving StateManager in my next post.

My first Android app...

Last night I successfully implemented my first Android application. I wrote app which will place a call to my wife's mobile. I named it 'Call My Wife' :). Now I can call my wife with one tap. Though it just little more than Hello World sample, its fun and exiting to get something end to end done and also to see some thing I developed running on my mobile.

Its a small first step and I hope I will be able publish my own application in Android market soon. I thinking of building some application using my expierence in Indian Language Technology.

Work on move... or coding on mobile

One thing that amazed me recently is the advancements in mobile world and the speed of these innovations. Especially the android market. The first mobile that I got had a small screen and a 2D snake game. It was just a phone!! Three yrs back when I got a walkman series phone I loved the quality of music it offered. After 3yrs I thought of buying a new one and confused... so many features in little phone...finally got a good smart phone. And ofcourse android.
Recently I read review of a phone with 2 screens and these can be combined as one screen in tablet mode!! Looking at these1GHZ and 32GB+ mobiles I think its not long away before we can start  coding and test on a mobile itself. We might soon get eclipse for mobile platform and SDF. We already have oracle lite, database that can be installed on devices. May be we can say "work on move" instead of "work at home".
(I am posting this from my mobile using google's blogger android app)

10 Years on... Java Coding and Akshara...

Its almost 10 years that I started writing Java Code. I like relook at the begining and introduce my first project/product I worked on.
Its 2001 January I started my 6th Sem Project as part of my MCA (in University Of Hyderabad) course. I decided to work on a module of Multilingual (Indian Languages) text editor along with my classmate Ganesh under Prof K Narayana Murthy. This project is executed as part of TRCT (Technology Resource Center for Telugu) in University. This is one of the 11 reaserch centers/organizations setup for developing Tools and Technologies for Telugu. Being open source and to achieve platform independency it was decided to develop this product using Java. This is why and how it (Java coding) started. No prior or formal training!! Started with Java1.1 but soon moved to 1.2. Person working on this product had to leave and we had to take over complete product and this happened around Feb 2001. We got first basic version done in couple of months... but was slow (on 216 and 512MB machines). So we had re-design basic things and code. The next version came out impressive and promising. It was named "Akshara" - Advanced Text Processor for Indian Languages.
Project got over in July 2001. I was supposed to join in my first Job in SQL Star International but thanks to rcession joining date was delayed..... Luckily I was offered Job in same organization I did my project and to work on same product along with 2 of my classmates. So in 1st Aug 2001 I started coding for Money... For next two and Half I continued there and during that time we added more features to Akshara and also created a chat tool and a Technology for web input and output along with other things.

After many years and many projects Akshara still remains my best work so far from personal satisfaction factor. Below are few features of Akshara.

• Open sorce
• Platform Independent
• 14 Indian Language support
• Basic Text editing featured and styling (Alignments, text styles etc)
• Uses own and open format (XDL) for storing files
• Embeded Email Client
• Spell checker
• Text to Speech
• HTML file creation with CSS

References to Akshara can be found at below links

http://www.lisindia.net/Telugu/telugu_Tech.html
http://tdil.mit.gov.in/TDIL-OCT-2003/text%20editors%20&%20word%20processors.pdf

Using Re-Use...

As discussed in last post re-using existing code/components is one of the key thing for better development. Every organization will have repository of reusable components but how do we ensure their re usability? From my recent observations things that can be reused are

1. Common functionality exposed as services
2. Validation and other similar utilities
3. Edit rules and other UI logic

One stage where we can identify reuse is during design phase. You can easily identify the reusable services/components at this stage, But you can identify reusable validation utilities and other reuse opportunities only if you undertake detailed/low level design. But how many times we perform low level design? In these cases onus of identifying the reuse lies with developers and development leads. But bigger challenge here how will these guys know that there is some thing that can be easily reused? As a simple example a developer who is required to do date validations will know that there is a Date validation utils already written by some one else? In general hectic schedules it is very much possible that we oversee reuse opportunities and end up coding again (reinvent wheel).

The thumb rule that I like to propose for developers is... while coding if you happen to code any thing that is not very close to Business of the application, then there is an opportunity for reuse. For example if you are coding a rule like - calculations based on some combinations of values selected on UI and some algorithm specific to application then there wont be much reuse opportunity. But if you are coding some Date validations, UI edit rules using Javascript (like restricting text field to accept only numbers) there is chance for reusing some thing existing. If you observe these are independent and had no close relationship with any business logic. In these kind of cases, you can search your organization's repository to find out suitable utilities.

I will continue discussing this topic with specific examples in further posts.

Recently I read some overview of Language called Scala which promises excellent support for component development. And it says Software development will become similar to assembling hardware. Means you will just need to put together some components and get the final product. This is an interesting thought and planning to spend some time on this.

A Good Developer

What it makes to be a good developer? This question keeps popping now and then in my mind especially when I hear complaints from managers and leads saying their developers are not good enough. In Sun Tech Days couple of years back, there was a session according to which below are the 7 best habits of a good developer.

1. Understanding the Problem
2. Use appropriate tool
3. Strive for Simplicity
4. Keep your code clean
5. Learn to debug
6. Leverage whatever available
7. Plan ahead

Lets discuss these in the context of general projects in Indian service industry

1. Understand requirements. This doesn't mean developer must have domain knowledge and understand business requirements but at least they must understand what should be done on click of Submit, click on Cancel, what formatting and validations need be done etc

2. Tools should have been decided before development start so lets not dwell much on this

3. 'Think Simple' - I keep telling my developers. Your performance is not measured based on number of lines you code or how complex your logic is.

4. Learn to code lean and clean code. This is possible only when following coding standards becomes a practice rather than compulsion. Also developer should emphasize on learning programming than programming language. I don't mean we don't have to learn any language but if you don't know how to solve a problem and write a basic algorithm there is no use of becoming an expert in Syntax of any language. Again you don't have to solve complex issues. Take for example age validations, if you don't know how to calculate age is there any use even if you know whole Date API in Java? Also emphasis more and more going in getting things done than on how? If developer gets a function point implemented that's enough, we rarely see leads reviewing for algorithm or developers caring to analyze their logic. No one to blame because of time lines but if you want to become a good developer your logic and its performance does matter a lot. Remember the Algorithmic subject thought in college and the order of executions?

5. Debug the code - No one writes a perfect code in one shot. With the latest IDEs and plug-ins debugging is easy than coding.

6. Leverage whatever available - Most of the things we do these days we are not the first ones doing them (Unless you are lucky like me). Refer organization's reusable repository, colleagues experience or online community to get the code. God had given us Google and a brain to use it.
7. No need to stress about time lines... Think and plan ahead. One of my professor used to emphasize to have complete algorithm written on paper and analyzed before even writing single line of code. Plan should include similar thing.. plan what to be coded and how to so that you wont end up with rework or stuck in middle.

To summarize, to become a good developer first thing to learn is problem solving (say analytical skills or whatever). Learn from all possible sources (read open source code, online material etc).

As Brian Kernighan and Dennis Ritchie say.. "only way to learn programming language by programing in it".

Any training can teach you API but not thinking.....