My ADTL Project

Hi friends. It’s been quite a while since my last post. Hope you all are doing well! 🙂

This post discusses my ADTL project, what all we did in it and the tools we used. ADTL stands for Application Development Tool Laboratory (It was only while writing this post did I check out the full form. Never bothered actually! :-P). Under this, we are supposed to select a problem statement from any domain. Use any tools or languages and create some application during the course of the semester.

The topic we chose was “Personalized News Reader“. It is basically an application which uses the pages you have liked on Facebook OR the people/organizations you are following on Twitter and then generates personalized news depending on your interests.

The idea came from the Hackathon I had attended recently, organized by Pune Google Developers Group. One of the participants and now a good friend of mine, Virendra Rajput had made an app which used Facebook likes to generate TV Serial suggestions. I liked the concept and the way they implemented it. And using Facebook API’s is something cool, isn’t it?

I suggested this idea to my project partners – Vaibhav Chandak and Vibhav Bhave and they too liked it. We quickly decided what we were going to do exactly and finalized the topic.

So there we were. Topic decided. No one knows whats the road ahead. How to get the page likes? How to get the followers? Where to get the news from? Which language is preferable? Which database to use? And many more….

Thankfully, our project was quite modular. It consisted of smaller modules which could be tested individually and then integrated with some simple linking of modules! So we decided to do it step by step and this is how we proceeded.

First, we focussed on the Facebook part. We decided to go for PHP, though none of us knew a shit about the language! 😛 But the API’s available are good and popular for PHP and the power of the language itself is what made us go for it and also we would learn a new language by the end of the project. The API we used was the Facebook API for PHP. For testing how the API works, we used the Graph API explorer provided by Facebook. It is similar to the recent graph search. Just the results are returned in JSON string format 🙂 To practice the API, I even wrote a script one fine day which downloaded all the profile pictures of my friends and saved them to disk in a directory! 😀 The Facebook module was prepared and it felt good to use the API’s. We also filtered the results for certain fixed categories. Pages like “Aap Dude Hain” and “Bahot bhookh lagi hai yaar, subah se kuch nahi khaya” have no meaning at all! 😀

Similarly, we also used the Twitter API for PHP. We used thmOAuth library by Matt Harris. It is a very nice library actually and handles OAuth very nicely compared to the Facebook API. We were able to retrieve the Twitter followers. But again, the same old issue. What about the ‘real’ friends whom we are following? Unfortunately, they are not so popular yet to get news for them 😦 😀 So we had to filter out these as well. For this purpose, we used the concept of verified accounts. We used only those followers whose accounts were verified by Twitter. And the second module was also prepared!

Next, before bothering about the news, we started making the Website. We felt that would give us an idea regarding the entire flow of events. This job was done in no time and we had a fairly decent UI in a days time 🙂

Now came the important part – Fetching the news for each of the filtered results. We had Google News API in our mind at the beginning. But the banner on their website dint please us. The API was deprecated! So we had to think of some other alternatives. I went on to Google API’s IRC and a guy called Jeremy helped me out with the problem. He suggested me the Google RSS Feeds API. You just have to feed a query of a particular format (like this) to the API and it gives your back results the same way Google News gives. So we explored the API and it was very easy! But again, a new problem arised. We had got results in PHP and this API was for JavaScript. Combining the two and mixing the server and client side functionality dint sound cool. So we searched for alternatives and finally decided to go with a function in PHP called as simplexml_load_file. This loads the query in an XML file and you just have to parse it. This has a disadvantage and an overhead which slows down things. But there was no other option to think about within that time. So we went ahead with it and were able to fetch and parse the news and display them with our own CSS.

But this was very slow. We needed a database where we could store the page likes and Twitter followers so that we can use those details when the user is signing into the app for the second time or later. As we knew only MySQL database, we decided to go with it! 😛 We started making the Database Schema and figured out in no time how complex it will happen. It would have had multiple levels of indexing and difficult to query. Furthermore, when the database grows, it would have turned slow.

The solution to above problem was Neo4j – a graph database. Actually, Neo4j was my seminar topic. I had to study the topic in depth and had to deliver a presentation on it. So, I figured out we could use Neo4j and it would be simple to implement it. When I told about this to Vibhav and Vaibhav, I remember the look on their faces 😛 Abhi kyun change karna hai yaar?? But as I had studied the topic, I knew how good it was and I managed to convince them 😛 They also studied it later and then we started off with the database part. We used Jadell’s library for this purpose. Again, an excellent one! It was fun to work on Neo4j where you could see everything in the form of graphs. For example, Pratik -> LIKES -> Sachin Tendulkar -> hasNEWS -> <the news retrieved for ST>. I was amazed by the graphical representation provided by them in the web console (See screenshot below). We went to Vibhav’s place on 18th, Thursday evening and implemented the database part staying up for the entire night. I went back to hostel at 6.30 am. Twitter part was covered but a slight problem was still there. I went back home at 9 am and started working again. Twitter part was completely functioning by afternoon and then it din’t take much time for FB part as the procedure was fairly similar. By evening 7 pm, I was done with the project and mailed the code to Vibhav and Vaibhav.

One problem which we faced was that of the McAfee Web Gateway in college and hostel. It blocked the API calls and we could not work at all. So most of the coding was done on weekends at home where there were no proxy issues. On the day of submission, we managed to get a no proxy port with the help of one of my senior and a good friend, Devwrat and used it for the project demo. Ma’am was satisfied with our implementation and also asked us to work more on it in the summer vacations. Adding new features and more personalization, etc. We gave a nod and were done with the project demo. It was satisfying at the end of the day.

I would like to thank all those who helped me during the duration of the entire project. Madhur and Rahul helped me finalize the topic. Rahul was even in constant touch with me during the project. Also, Harsh helped a lot when I had problems with some PHP code. So a big Thank You to you guys! Seniors ho toh tumhari tarah…

It was great to work on the latest techlogies – Facebook API, Twitter API, RSS Feeds, Neo4j
Lots of learnings and much more to come 🙂

Here are some of the screenshots of the application.

The Home Page

The Home Page of the Application

The Modal Window asking to chose between Facebook and Twitter Login

The Modal Window asking to chose between Facebook and Twitter Login

The News Page showing results one after the other using my Facebook likes

The News Page showing results one after the other using my Facebook likes

Neo4j Graph View

Neo4j Graph View – Pure Awesomeness!

Thank you for reading! Feel free to give any sort of feedback and suggestions to improve 🙂
Cheers and Happy Coding as always! 🙂

Advertisements

2 thoughts on “My ADTL Project

  1. Awesome job dude.
    It would have been good if you could have done it in Google App Engine, so that it would have been live, for everyone to use. We would have got the privilege to use this awesome thing.
    One more thing, you can work on in the future is improving the intelligence of the algorithm that finds the relevancy of google news on basis of the ‘likes’.
    But all in all its a great achievement, you have more on the ADTL than most work on the final year project, just kidding 😛

  2. Thanks a lot Rahul. I finished this entire thing just the day before the submission. So dint think about Google App Engine. But yes, I will definitely host it there and make it live sooner or later 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s