I recently worked on developing an all-new status update engine in the Open Source club management system(CMS) we were developing at our club – amFOSS. This article shares why status updates are important for us as a community, how we are implementing this system, and also talks about the limitations of the previous status update automation we had.
amFOSS (previously [email protected]) is one of India’s leading FOSS & Computer Science student-run community and was established back in 2007, at Amrita Vishwa Vidyapeetham, Amritapuri for the cause of promoting and contributing to Free and Open Source Software among college students, whilst achieving excellence in the field of technology in their pursuit of knowledge. The club has more than 50 active members today and has produced 50+ Google SoC students in the past 10 years, listing the university among the top 10 globally. Several of its alumni are presently working in Fortune 500 companies including Google, Facebook & Microsoft among others, while many are pursuing their masters and PhD. at various prestigious universities across the world.
What are Status Updates?
At amFOSS, every member is required to send status updates as an email reply to the thread generated daily summarising (or sometimes, explaining) the work they did that day, work they plan to do the next day and the blockers they might be facing, at the end of the day.
These are much similar to ‘daily standups’, ‘daily huddle’, ‘morning roll-call’ etc. that are commonly found in the industry.
Why Status Updates matter?
- Monitoring & Accountability: Since the status updates are send to a common thread everyday, and visible to all members of the club, they always come under the eyes of several senior members who can thus always keep an eye on a member’s progressive growth. The status updates are often the best source the mentors look up to find out what the member has been doing over a timeline, so as to share suggestions, feedback for them. Regularity in sending status updates is one of the factors that we have been using to judge a member’s commitment to the club as well.
- Improves Language & Communication Skills: We have members with diverse linguistic backgrounds at our club, many of whom lack enough proficiency in English. The practice of sending status updates daily help members to train and improve their English writing skills over a period of time. In fact, it also teaches them to write formal emails, summaries, work reports and introduces them to the ‘daily standup’ practice all of which are common in the industry.
- Increased Transparency & Trust: Being advocates of FOSS, transparency is something that we take seriously. The system of status updates being shared daily by all members on a common thread, helps each of us to know what other’s are doing. It also sometimes helps us to find others having similar interests, identify subject experts, and to get exposed to cool things that others have been doing.
- Receiving Feedback & Help: The status updates also act as a platform for members to shout the blockers they are facing so that they can potentially get help from any other member reading it. The senior members also sometimes send feedback individually to members such as general comments on language feedback, suggestions on the work they are doing, and recommendations for their future plans.
The need for an automated monitoring system
Managing and monitoring a fast-growing, diverse community of 50+ members has never been easy at the club. Being wholly run by the students, the process of decision making and its enforcement in the club to fellow students is most often challenging.
Despite for all good reasons the status updates system was introduced, naturally many of us often skip sending status updates daily, send it late, or just goof up something and send. Since it is not practical for someone to keep log everyday, the need for automated scripts was well-conceived when it was introduced.
The old system
The club’s django-based website used to run a cronjob everyday to send status update thread and fetch status updates via Gmail APIs, log them in the dB and send out a report in the club’s telegram group. The telegram group is our main platform of online discussion with all our members and faculty mentors including Vipin sir. The pressure of getting one’s name among “those who did not send status update today” in the group, brought most of the members to send status updates daily. But, it had a few limitations.
The limitations of the old system
Crossing the Deadline
We had initially kept a deadline to send the status updates at 4 AM daily. At 4 AM, the cron job used to log these, and reports used to get generated. However, it was found that many of the members were sending out status updates hours after midnight, which was concerning for us as this indicated they were losing sleeping and overkilling themselves. Soon, a decision was made to keep the deadline for the status updates at 11 pm right after the lab was closed, with an aim to curb people from working late night as “they might sleep earlier, maybe as they cannot account the work done after it in the status update”…
After implementing the system, we received numerous complaints as at the deadline many of them were packing up and heading back to the hostel, and the deadline had passed by the time they reached. Many of those who wanted to send was now not sending as it wouldn’t be anyway accounted for after the deadline had passed. Thus, we decided to allow accepting status updates till 4 AM, but mark those after 11 PM as late. This meant we needed to add an extra attribute to the logs, the timestamp of their mails.
No light on a person’s track record
Another hurdle we had was that though the list of people who did not send updates for the day was getting reported in the group, it showed little about the person’s history – has he/she not send for a couple of days now, whether this is has become a habit, how regular has the person been in the last month etc. Though all these could be manually queried from the dB anytime to generate a report, its absence in the report allowed people to send less frequently and get away without getting noticed. Thus, we now needed to also add their previous track record as well into the telegram message that was sent out to the group daily.
Controlling, Scaling & Maintaining
We sometimes needed to pause the system such as during vacations, and exam time. In the old system, we were using a hardcoded crontab in the server, which needs to be removed and added back to achieve pausing. However, this task couldn’t be assigned to many people – as this needed access to the server and required a good understanding of how it works. Most of the features, messages all were hard-coded, which didn’t allow us any customisations.
Also, we now wanted to run scripts and have separate threads for recruit batches (who were not yet in the club), internal teams etc. However, the old system could only support a single thread. Further, the old system took hardcoded parameters – thread email, telegram bot token, telegram group id, which all were stored in ‘server_settings.py’.
We, therefore, decided to build an all-new status update engine in the club’s new club management system(CMS). The new system must be able to support any number of threads, report out the past track record of the person, mark late updates and customise & control all this from an easy use UI without any need to touch the server.
In the next post, I will share how and what implemented the all-new engine and describe the new features it offers.
This article shares my experience working as the lead web developer at amFOSS, and as the core developer of CMS, while working on this project. The points shared here are my personal views, opinions and conclusions, and may not be consistent with the club’s official statements. This is not an officially endorsed/accepted blog post by amFOSS.