Can we discover buzz patterns from Blogs?

The huge Consumer Generated Media ( CGM ) or User Generated Content ( UGC ) available in terms of blogs, social networks , public Wikis and other Internet based content stores have always inspired me to find out the answer of the following question

Can they be used to determine a trend or buzz for a specific business entity ( ex. Products like Apple iPhone or any TV soap like The Prison Break) ?

A typical example is the following blog post on a restaurant named “Mainland China”
http://bangalore.metblogs.com/archives/2006/12/dining_out_mainland_china.phtml
and comments from lots of blogosphere users on that post,
Now the question is whether this small fragment of user generated content can act as a piece of "gyaan" which can be effectively searched and therefore seamlessly discovered on the Web?

When I searched google for "Mainland China Bangalore" or "Dining Out: Mainland China" the above mentioned page came up as one of the top results. Now this is something that the internet users do regularly , though among all the internet users only a few come up with good search phrases that ensure contextual results. The popular search engines cannot take into account the meta context information which can possibly be best defined by the intent with what the author has written a specific blog post. Possibly the title of the post can be taken as one important context for any keywords that we index from a blog.

All those important information posted by the bloggers and other CGM providers will effectively be lost if we don't bundle them with a specific context ( As. Tagging ) or make them searchable
( Indexing ).

Some of us have already seen the Blog Buzz implementations , these are typically user driven classification and categorization of blogs and other UGC for better presentation patterns in search result. But definitely we can improvise context driven classification of contents and search upto a point where we can provide the users with structured information like www.wikipedia.com provides.

I looked into existing blog search engines like

1)Technorati
2)Google blog search
3)Blog pulse and
4) "Nielsen BuzzMetrics(www.nielsenbuzzmetrics.com)"

All these sites index blogs and provide search interfaces on them , when some of them has gone one step ahead in providing structured trend information out of the blog content. But looks like they have a long way to go.

Basically my idea is to come up with a very basic implementation that does the following

1)Crawl a set of blogs belonging to a very specific domain ( Ex. Restaurant or Movies )
2)Index them in the order of business entities they primarily talk about.
3)Present the information in a review oriented format on a brief , user friendly UI.

Mama said "Days are going to be tougher"

This post is written in response to a recent blog about a major IT company based in Chennai and Hyderabad.

I think most of us, who have worked in Company X at some point of time have faced all the issues and inconveniences described in that blog and yes most of us have stated this phrase myriad times ...”Company X sucks dude”. My stint with Company X lasted for two long years and currently I am in Bangalore, working for one of the major dot com companies. I started my term in Company X during a time in Company X’s history when accessing proxies from local desktops was allowed, so as free internet access, we used get one and a half thousand bucks worth gift coupons on our birthdays, it was a very small crowd then and yes we used to feel great about the three and half lakhs fresher’s pay package that we used to draw.

But all of a sudden it started happening, free access scenarios were being banned, people got fired due to unethical usage of internet, submitting spurious reimbursement bills, misbehaving with the opposite sex etc. etc. Company X faced a huge exodus of the employees and attrition rate raised to the all time high but that was the time for mass recruitment too, it partnered in almost every job fair happening in and around Chennai, its name started coming in the print media frequently because of the huge waiting queues it had outside its recruitment events. So that small Company we were working wasn’t miniature anymore, it became a full grown monsterJ. In course of time its India operation’s headcount was numbered around 3-4 thousands and it started operating from two metros Chennai and Hyderabad.

Now if you look at the demographic of projects that Company X typically does, 50-60% of it is technical support, 30% maintenance and a staggering 10-20% is fresh development. And guys lets not be sentimental here while keeping in mind that this India operation was established due to Company X’s pursuit to cut down its operating cost profusely and that’s the way it is. But as you know that back in US Company X was already suffering from a number of legal obligations and the relation between Company X and FCC was on the rocks, all these required Company X to establish its offshore operation under strict legal clearances (I know several projects never came to India because of the same). Some of these legal issues were so intense that they could have nipped X’s India operation in its bud. So the mandate of the higher leaders was something like “let’s take the minimum risk in legal terms and make our security norms bullet proof”. When you run a technical support centre you hardly want that to come up with innovations and this philosophy in turn reduces the need of the exposure to the information broadways like internet for the employees. But I think you still have a lot to learn and perform in Company X, in terms of the materials that are available in it’s intranet, research documents by leading strategic management firms like Gartner, free access to premium online libraries, access to the code base of some of the industries’ most innovative products and others behavioral aspects like open culture and allowance of ownership taking .So end of the day it is not only a place of have-nots.

I would like to take this opportunity to point out some of my concerns on the overall corporate culture that I have observed in Indian IT industry, which is primarily driven by the outsourcing boom.

Are we learning how to separate out our personal and professional lives? Why we still look for friends, lunch mates or dinner mates or some other “mates” at work but not try to treat people as “only colleagues”?

Why we end up staying much more hours in office than we need to finish our work items?

Why don’t we measure a job offer properly in the scale of our own personal priorities and future plans? Sounds pretty awkward to me that someone’s job is stopping her to become an ideal wife …..

Why we want to listen to pirated music in our company desktops when we know that it can lead the entire organization towards huge obligations?

Why we confuse important organizational security measures as policing in our own territory?

Why someone has to feel like being “caught by a cop for drunk and driving” when a security guy asks him to show his id card?

Aren’t some of these physical securities meant for our own benefit?

Why the hell we want to bring CDs, DVDs, and USB Drives to the work? Please do a reality check on how many of you actually own non-pirated DVDs and virus free USB Drives?

Why some of us come to office on weekends only to enjoy the air condition and free Internet?

And finally I think the rule of thumb is, if you think your workplace is making you unhappy, you better quit the job and not crib like a premature …as a company has other things to consider than fostering a bunch of gizmo freaks. It’s sad but true …. I have observed that some of us even take tissue papers to home from office to save a tiny amount of personal stationary cost. This is just ridiculous, isn’t it.

Personally I would request people to be more positive in terms of understanding work culture which is very specific to a particular organization. We are burgeoning as the largest IT labor pool and we can support this growth to sustain definitely not by abusing leaders and other concerned people or expressing qualms against what we are getting end of the day but by sharing positive feedbacks and helping leaders to implement robust processes, which will finally benefit ourselves creating a better ecosystem in which we all are destined to coexist.

Ypodia! – The yahoo search experience in the wikipedia way…...

Vipin asked me to define the problem …which was indeed an intimidating demand of his. Yes, how about user tuned web search or sharing the search query strings (on specific keyword) on the web, well as the web user count is increasing each and every day there is a increased need to mirror the searches that has happened on a specific keyword and sharing the web searches made by experienced web user to the newbie …isn’t that idea comprehensive enough. My idea is to have a wiki, where instead of static articles...It will contain links to different information sources grouped by different contexts and different search APIs like (Google, yahoo, technorati search and yahoo answers .Flicker photos etc.)

Introduction:

Web search has become the hottest application on the wire in recent years, but the online encyclopedia tools like wikipedia still prove their success over powerful web based search engines like Yahoo or Google when a user wants to find some specific information on the web with search keywords, like “second world war” or “Microsoft corporation”. Just to give an illustration, if you search Google or yahoo for “Microsoft corporation” will return you links from web like the following ( top 10 results )

1) www.microsoft.com

2) en.wikipedia.org/wiki/Microsoft

3) support.microsoft.com

4) home.microsoft.com

5) msdn2.microsoft.com

6) office.microsoft.com

7) office.microsoft.com/en-us/frontpage/default.aspx

8) www.research.microsoft.com

9) msdn.microsoft.com/xml

10)members.microsoft.com/careers/default.mspx

Now, lets see what happens when we go to wikipedia for the same

It takes us to the page of http://en.wikipedia.org/wiki/Microsoft_Corporation article, which comprises the following

* 1 History

o 1.1 1975–1985: The founding of Microsoft

o 1.2 1985–1991: The rise and fall of OS/2

o 1.3 1992–1995: Domination of the corporate market

o 1.4 1995–1999: Foray into the Web and other ventures

o 1.5 2000–2005: Legal issues, XP, and .NET

o 1.6 2005–2007: The road to Vista

* 2 Product divisions

o 2.1 Microsoft Platform Products and Services Divisions

o 2.2 Microsoft Business Division

o 2.3 Microsoft Entertainment and Devices Division

* 3 Business culture

* 4 User culture

* 5 Corporate affairs

o 5.1 Corporate structure

o 5.2 Stock

o 5.3 Diversity

o 5.4 Logos and slogans

* 6 Criticism

o 6.1 Corporate

o 6.2 Technical

* 7 Microsoft.com

* 8 See also

* 9 References and footnotes

* 10 External links

Isn’t that great, a single source for all your want to knows … you can know better about Microsoft corporation from this page than anything available on the web, as this is a very structured and globally contributed article which pretty much cover the entire Microsoft story, so now on you will always go to this page instead of doing popular search engine search.

But lets not stop here, lets take a third approach..What happens if I would have searched web using the globally shared contexts on “Microsoft Corporation”.

Like the following

search string
(contexts)

a)Microsoft corporation history:


##### results #### ##### source #####

en.wikipedia.org/wiki/Microsoft yahoo search
www.radessays.com/viewpaper/13230/Microsoft_History.html yahoo search
www.microsoft.com/billgates/bio.asp yahoo search
www.answers.com/topic/microsoft yahoo search
www.thocp.net/companies/microsoft/microsoft_company.htm yahoo search
www.csl.mtu.edu/winter98/cs320/micro/history yahoo search

The history of Microsoft Corporation? yahoo Answers search
The History and Development of Microsoft Corporation? yahoo Answers search


b) Microsoft products


www.microsoft.com/products yahoo search
www.microsoftproducts.net yahoo search
technet.microsoft.com yahoo search
office.microsoft.com yahoo search
msdn.microsoft.com/canada/academic/products yahoo search
members.microsoft.com/careers/careerpath/
marketing/product.mspx yahoo search
tech.msn.com/guides/msproducts/default.aspx yahoo search

Why do Microsoft products suck? yahoo Answer search
Why do some people hate Microsoft, but still use
Microsoft products? yahoo Answer search
Microsoft products? yahoo Answer search
IBM and Microsoft product compatibility? yahoo Answer search
How can I find who a Microsoft product is registered to? yahoo Answer search
Stupid Microsoft Product Activation Bullshit problem? yahoo Answer search

Now think of a wiki, which contains a log if all the searches made on specific keywords with different contexts. A place where you will find a collection of links grouped by contexts applied on the search keyword ( those linked are typically fetched by searching across the web using popular search apis like Yahoo , Google and Yahoo answer , flicker …basically a detailed snapshot of all the search query combinations available for the specific keyword ).

Gory Details:

How it works? Possibly that will an interesting question to answer at this point…

This is a new type of wiki driven by search , as said that basically creating a document here is something like this , you define a keyword on which you also create search contexts like if the keyword is a person , as Shahrukh Khan , the possible contexts will be “filmography” , “awards and nominations” , “scandals” . Then once you define the contexts, you will be given the options to choose the search channels , like Yahoo search , yahoo answers , de.icio.us tag search , technorati blog search and when you select the search channels and generate the results …you will be given control to choose the results. Now once you choose the results …it will wiki the whole thing, keyword as the document root …under that various contexts defined and under each context the links that you have chosen. Now when I say that the whole thing is going to be built on the wikipedia infrastructure …then its valid that anyone can add more contexts and refine and redefine any of the existing contexts.

Eats-shits-and-leaves-of-my-life

You know what ! I committed this crime once again , another weekend ( long one as they say ...when we have three days of retire) and me in my favorite apartment with my laptop .. 250 GB load of movies ..so Sunday became a nice movie day for me! Watched 5 movies back to back! ( without a single grain of pop-corn) ..the one about which I want to talk about is “ Ask the Dust” , a Colin Farrell and Salma Hayek starrer . This was the movie I was waiting for a long time , because its about a writer who looks at people , their life and try writing things from his experiences. Arturo Bandini , the hero of the movie , is a writer by profession who is trying to put himself at a good place in American Literature. He moves to LA with the aspiration of look at the life of its ( LA ) population , the hotels , restaurants , parks , pubs , malls , streets , ghettos everything. Now , as the money left in his stash is fast effacing , he faces a real tough situation to maintain his livelihood in LA , to pay his rent , get a square meal. But he doesn't give up and he continues to do what he was doing ...sometime wrapping tobacco with toilet paper for his joint , biting the sour oranges he manages to get from the Chinese cheap fruit seller ...continues his battle till the day when is left with a nickel ...wants to grab some coffee before being bankrupt ..which lead to meet this Camelia Lopez ( Salma plays that character ) gal , a sweet Mexican girl who is the waitress in a bar and incodently she serves him some real bad coffee ...worst of its kind to spend one's last nickels on ...gets into a fight with her and falls deeply in 'Love' ( well ....I don't claim that I know the meaning of this little word ..so you can read it as the “symbolic cynical chemistry in a man's mind for a woman” )...

Now there is another important character in this movie Vera Rivkin , a Jews girl who is a fan of Bandini's column , somehow comes to meet her favorite author Bandini , who can solve her problem ...all she wanted Bandini to say that “She is beautiful” ....finally she becomes a pivotal inspiration for Bandini to start on his new novel , which eventually fetches good amount of money and fame for Bandini. Now how the Camelia and Bandini's love towards each other grows and matures and finally how the movie end should be untold here to make sure that you watch it. But I would say , with my limited knowledge about woman ...Camelia , though portrayed as a rough-tough beautiful Mexican gurl ..represents those of its kind where they breath on their submission towards a meaningful relationship and expectation from the same. Bandini , well what you can think of a nice normal human , scared to get settled , confused about saying “yes” to Camelia ,whom he loves more than anything in this world, when she asked him to marry her , quite typical of a man character , his arrogance is marked by his uncomfortable way of treating a woman ..which I really found something unusual in Hollywood heroes.