Hi friends. It’s been quite a while since my last post. Hope you all are doing well! 🙂
This post discusses my ADTL project, what all we did in it and the tools we used. ADTL stands for Application Development Tool Laboratory (It was only while writing this post did I check out the full form. Never bothered actually! :-P). Under this, we are supposed to select a problem statement from any domain. Use any tools or languages and create some application during the course of the semester.
The topic we chose was “Personalized News Reader“. It is basically an application which uses the pages you have liked on Facebook OR the people/organizations you are following on Twitter and then generates personalized news depending on your interests.
The idea came from the Hackathon I had attended recently, organized by Pune Google Developers Group. One of the participants and now a good friend of mine, Virendra Rajput had made an app which used Facebook likes to generate TV Serial suggestions. I liked the concept and the way they implemented it. And using Facebook API’s is something cool, isn’t it?
I suggested this idea to my project partners – Vaibhav Chandak and Vibhav Bhave and they too liked it. We quickly decided what we were going to do exactly and finalized the topic.
So there we were. Topic decided. No one knows whats the road ahead. How to get the page likes? How to get the followers? Where to get the news from? Which language is preferable? Which database to use? And many more….
Thankfully, our project was quite modular. It consisted of smaller modules which could be tested individually and then integrated with some simple linking of modules! So we decided to do it step by step and this is how we proceeded.
First, we focussed on the Facebook part. We decided to go for PHP, though none of us knew a shit about the language! 😛 But the API’s available are good and popular for PHP and the power of the language itself is what made us go for it and also we would learn a new language by the end of the project. The API we used was the Facebook API for PHP. For testing how the API works, we used the Graph API explorer provided by Facebook. It is similar to the recent graph search. Just the results are returned in JSON string format 🙂 To practice the API, I even wrote a script one fine day which downloaded all the profile pictures of my friends and saved them to disk in a directory! 😀 The Facebook module was prepared and it felt good to use the API’s. We also filtered the results for certain fixed categories. Pages like “Aap Dude Hain” and “Bahot bhookh lagi hai yaar, subah se kuch nahi khaya” have no meaning at all! 😀
Similarly, we also used the Twitter API for PHP. We used thmOAuth library by Matt Harris. It is a very nice library actually and handles OAuth very nicely compared to the Facebook API. We were able to retrieve the Twitter followers. But again, the same old issue. What about the ‘real’ friends whom we are following? Unfortunately, they are not so popular yet to get news for them 😦 😀 So we had to filter out these as well. For this purpose, we used the concept of verified accounts. We used only those followers whose accounts were verified by Twitter. And the second module was also prepared!
Next, before bothering about the news, we started making the Website. We felt that would give us an idea regarding the entire flow of events. This job was done in no time and we had a fairly decent UI in a days time 🙂
But this was very slow. We needed a database where we could store the page likes and Twitter followers so that we can use those details when the user is signing into the app for the second time or later. As we knew only MySQL database, we decided to go with it! 😛 We started making the Database Schema and figured out in no time how complex it will happen. It would have had multiple levels of indexing and difficult to query. Furthermore, when the database grows, it would have turned slow.
The solution to above problem was Neo4j – a graph database. Actually, Neo4j was my seminar topic. I had to study the topic in depth and had to deliver a presentation on it. So, I figured out we could use Neo4j and it would be simple to implement it. When I told about this to Vibhav and Vaibhav, I remember the look on their faces 😛 Abhi kyun change karna hai yaar?? But as I had studied the topic, I knew how good it was and I managed to convince them 😛 They also studied it later and then we started off with the database part. We used Jadell’s library for this purpose. Again, an excellent one! It was fun to work on Neo4j where you could see everything in the form of graphs. For example, Pratik -> LIKES -> Sachin Tendulkar -> hasNEWS -> <the news retrieved for ST>. I was amazed by the graphical representation provided by them in the web console (See screenshot below). We went to Vibhav’s place on 18th, Thursday evening and implemented the database part staying up for the entire night. I went back to hostel at 6.30 am. Twitter part was covered but a slight problem was still there. I went back home at 9 am and started working again. Twitter part was completely functioning by afternoon and then it din’t take much time for FB part as the procedure was fairly similar. By evening 7 pm, I was done with the project and mailed the code to Vibhav and Vaibhav.
One problem which we faced was that of the McAfee Web Gateway in college and hostel. It blocked the API calls and we could not work at all. So most of the coding was done on weekends at home where there were no proxy issues. On the day of submission, we managed to get a no proxy port with the help of one of my senior and a good friend, Devwrat and used it for the project demo. Ma’am was satisfied with our implementation and also asked us to work more on it in the summer vacations. Adding new features and more personalization, etc. We gave a nod and were done with the project demo. It was satisfying at the end of the day.
I would like to thank all those who helped me during the duration of the entire project. Madhur and Rahul helped me finalize the topic. Rahul was even in constant touch with me during the project. Also, Harsh helped a lot when I had problems with some PHP code. So a big Thank You to you guys! Seniors ho toh tumhari tarah…
It was great to work on the latest techlogies – Facebook API, Twitter API, RSS Feeds, Neo4j
Lots of learnings and much more to come 🙂
Here are some of the screenshots of the application.
Thank you for reading! Feel free to give any sort of feedback and suggestions to improve 🙂
Cheers and Happy Coding as always! 🙂