SETTING THE RIGHT EXPECTATIONS

This is not my typical technical article, just a little restrospective of an experience at work.

I was working with a new member in the team, to provide him the KT on one of our applications. I asked him to pull our source code from git, and he told me that he has never heard of git before. I showed him a few basic git commands and asked him to to set up the code locally. Once he got stuck after a point, he asked for my help. I asked him where he had checked out the source code and he pointed me to his desktop. I found a highly cluttered and unorganized desktop, and the source code of our project was right in the middle of that mess.

That immediately turned me off.
Soon I realized that he had no prior experience with many of the tools and software that we use largely at work. I got pissed some more.
After a while, I went to my supervisor and asked him who had interviewed this candidate for our project. He told me that he had no idea who interviewed him and he was just assigned to our project to be added anywhere we see fit. I was appalled since he was an obvious misfit and had no basic knowledge of things we use on a daily basis. Is it a fair expectation and good use of my time to start training him on basics, which could have been easily filtered during the interview.

Patience is not my strong suite, but I have to play best with the cards that I have been dealt with. But may a times I keep questioning myself "What am I doing? Is this what I wanted to do?" The answer is yes, this is where I exactly wanted to be when I was a starry eyed kid who wanted to be a computer engineer. I chose this. But then I believed I would be probably changing the world with my work.

Like all people born in the 80s, I grew up seeing my parents working the same job day in and day out till they retired. They treated their jobs as jobs, worked like clock work, it wasn't meant to be satisfying, they didn't wish to change the world. They had secure jobs, smaller needs, secure retirements thanks to pensions, and ran through life on auto pilot. Now that they have retired, they spend time adding to the TRP of random TV serials, gossiping about neighbors and relatives, visiting sacred pilgrims and the worst sending useless forwards on WhatsApp and Facebook.

Agreed that this generation has got into easy jobs, I earn more in a day than what my father earned in a month, I bought a car and a house at an age that my parents couldn't even dream of buying when they were at the same age. But our needs have also consistently increased. We don't have secure jobs, no assurance of a pension in our sunset years, the added temptation of the phone on sale on Amazon or order the over-priced pizza for dinner, and a peer pressure to prove that we are better off than what our friends show off on Facebook and Instagram. While I do want to go on a foreign holiday like my friends on Facebook (even if they probably took a personal loan for it), I still lose my hair over my retirement savings plan.

But that is not a bigger problem for us today. The problem at hand is we crave job satisfaction. Many of us entered the workforce with starry eyes with the hope that we were here to make a difference or change the world, we have realized over the years that the most jobs are just soul suckers. Even if I cant make the world turn around with my work, I at least hoped to make a dent or leave a mark somewhere. If you don't own your own startup by the time you reach forties, you feel like a loser, if you do not look forward to your day at work every morning and rather are dragging your feet to work every day, its depressing. You try to shake things up, change things, inspire others while trying to stay self-motivated, or best keep job-hopping for every couple of years seeking gratification, and eventually give in to "leading lives of quiet desperation".

So now what!!! An early retirement. But retire to what!!!
I wouldn't know what to do with myself if I had no work for a week, what do I do post retirement. Scrolling through social media all day...how long can you do that!!! Watch TV serials all day...I don't think so. Spend time with kids...if they have time for you (if you manage to have one that is). Holidaying...needs money. Spend time gossiping with random people..Argh!!!
And what about the financial security. Without that pension, we are on our own. So we keep working, to be able to buy that fancy phone, or post pictures of our good times at a foreign holiday on Instagram, to add to our retirement savings if God forbid we live to see our 90s.
So it's important that we accept that while jobs are not necessarily meant to suck, they aren't going to provide any life changing experience either. Lower your expectations and you probably will not feel like you are dragging your heels through the day.

With that thought in mind, I will step into office tomorrow and continue my KT as planned. I probably won't be able to get the best of my expectations from the new guy, but I will try to make him the best version of himself.

Beyond CAPTCHA

If you have ever filled an online form, you are bound to have encountered a CAPTCHA.

CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. I haws been named after the computer scientist Alan Turing.

What is a Turing Test?
Turing test is used by humans to see if a machine can converse like a human. The turing test has evolved to an interesting concept. Today, its a scenario in which a human interrogator uses some sort of interface, often a computer terminal, to communicate with someone else. This some one else could be a another computer program or a human being.If a computer program manages to fool most of the human interrogators into believing that it's a human, the program is said to have the turing test.

The CAPTCHA test is a turing test designed as a short hand for a gateway that allows humans to pass through while keeping machines at the border. The idea of a CAPTCHA is  to build something which is simple for a human being to decipher while difficult for a machine to do so. This is important because it does not do any good if it discourages humans to fill out the said firm.
There are a few things a software can do better than humans, like driving a car or playing a game of chess under normal circumstances, but there are still things that humans can do better, at least that's how things stand today.

A simple CAPTCHA example the case where you have deformed letters and numbers semi-obscured by other shapes. While humans can easily differentiate these characters from the distractions, computer programs have a tough time segregating the signals from the noise. 

Evolutionary speaking, while AI keeps evolving all the time, humans have kind of sowed down in comparison. For ex: Researchers at the AI company Vicarious may have just made CAPTCHAs obsolete, however, by creating a machine-learning algorithm that mimics the human brain. They have created a artificial neural network that the firm claims can solve CAPTCHA puzzles by simulating the human capacity of "common sense". It works on a neural network that can suss out the shape of letters and numbers while while filtering out the distracting noise, by mimicking how a human brain works.
The artificial neural network attacks the problem in layers with each neuron attempting a part of the problem and then combining all solutions to come up with complete solution of the CAPTCHA. Solving CAPTCHAs is not the goal of the research, but it provides insight into how our brains work and how computers can replicate it.

According to security experts, we can expect to see hackers to make use of these technologies in a few months. So CAPTACHAs will become less effective in keeping the bots out.
As AI keeps getting better at  solving the CAPTCHAs, designers are coming up with more and more creative ways and these affect both humans and non-humans.
For example some sites have a small passage followed by a simple question about the content matter to test if the attempt was made by a human.

12-Factor Apps in Layman's terms

Popular platform-as-a-service provider Heroku has published a set of best practices called the Twelve-factor-app for building deployable sowtware-as-a-service apps that are

• Portable
• Continually Deployable and
• Scalable

Here are the twelve principles in a more precise and understandable version:

CODEBASE: One codebase tracked by revision control, many deploys 

Put all your code in a version control system, the best option for me is the git.

You can have multiple branches but never multiple repositories for the same service/ microservice. There can be only one codebase per app, but there can be many deployments. 

DEPENDENCIES: Explicitly declare and isolate dependencies

This has been more important ever since configuration tools like Chef and Puppet have been used extensively used by DevOps to automate the configuration and deployment process.

Consider maven as dependency management tool, manifest will be pom.xml, which fetches dependencies as jar (artifacts) from various repositories.

All your dependencies, third party or otherwise must be declared and completely isolated across environments. Most of the modern programming languages have a built-in support for this. You can declare all libraries in your code, and during the deployment pass the environment variable as a part of the command line that picks up the correct file to deploy

CONFIG: Store config in the environment

Configuration are those part of the application that change based on the environment where the application is deployed on. For example database connection properties, application specific information such as host IP and port etc.

There should be a strict separation between config and code. Code should remain the same irrespective of where the application is being deployed, but configurations can vary.

BACKING SERVICES: Treat backing services as attached resources

Backing services refer to the infrastructure and other services by which the application communicates over the network. Database, Message Brokers, other API-accessible consumer services such as Authorization Service, Twitter, GitHub etc., are loosely coupled with the application and treat them as resource.

The idea is to be able to swap any of these resources with a different provider without having to make any changes to the source code.

BUILD, RELEASE, RUN: Strictly separate Build and Run stages

The twelve-factor-app suggests strict separation of the build, release and run stages.
Build - converting source code into an executable bundle known as the build. (jar, war, ear etc.)
Release - Or Deployment. Getting the build and combining it with a configurations of the specific  environment, assign it a unique release number, and made ready to run.
Run - Execute the package on the specific environment.

The 'build'cycle does most of the heavy lifting and the run stage should be very light weight. Tools that help in achieving a full CI/CD pipeline are Jenkins, Thoughtworks go, and Codeship to name a few.

PROCESSES: Execute the app as one or more stateless processes

Modern applications are deployed on many nodes usually with a load balancer to direct the traffic to enable quick request handling. In such cases, we cant guarantee that consecutive  requests from the same client would go to the same node.  Hence its unwise, to rely on data stored in the previous requests, since it would not be available if the next request is directed to another node.
As a rule, you want each of those instances of running code to be stateless.

There are a number of ways to achieve this:

• Save the state of the process in the database or shared storage.
• Use scalable cache storage like Memcahe or Redis.
• Package assets in executables (e.g. by using webjars at build time)

PORT BINDING: Export services via port binding

This is an extension of  the Backing Service factor.
Make sure  that your application is visible to others via port binding, so that other services can use it like a resource.

CONCURRENCY: Scale out via the process model

This is all about scalability. The idea with more smarter applications is to scale horizontally by deploying more copies of our application on multiple nodes rather than scaling vertically, i.e running a single instance on a much powerful system.

In particular, you’ll be able to do more stuff concurrently, by smoothly adding additional servers, or additional CPU/RAM and taking full advantage of it through the use of more of these small, independent processes.

DISPOSABILITY: Maximize robustness with fast startup and graceful shutdown`

Processes in twelve-factor-apps should start and stop in minimal time.

Fast startup is based on our idea of scalability and also gives way to the usage of microservices as opposed to monolithic applications. If an application takes 30 seconds to start up and handle requests, it defeats the idea of rapid releases.

Graceful shutdown also involves leaving the system in the correct state. All resources should be freed and all unfinished tasks should be returned to queue. Crashes also should be handled. Crashes, if they happen, the app should be start back up cleanly.

DEV/ PROD PARITY: Keep development, staging, and production as similar as possible 

In the recent years, its become pertinent to have short and rapid production cycles. This means that the time window between the development of a change and deployment of the said change is become very small, sometimes a matter of hours. With this window being so small, you do not want to spend time on the deployment process, or carry the risk of something breaking in production.

Hence, its highly recommended to keep the developers environment as similar to the production as possible. This includes using the same third party(or otherwise) services, same configuration management tools, same softwares and libraries.

LOGS: Treat logs as event streams

Logs are highly useful. They can carry variety of information on many levels. They can come extremely handy while trouble shooting the errors in production. Read : Logging the right way

Other than trouble shooting customer issues, you can use error monitoring service like  Airbrake or Papertrail or data mining services like Logstash or Splunk.

ADMIN PROCESSES: Run admin/management tasks as one-off processes

Your app might need some admin jobs at times, for example cleaning up the database for bad data, switching on/off configurations or features, generating reports for analytics or audit purposes.

The twelve factor app suggests that you do these one off tasks from an identical environment in production. Don''t directly access the database, do not access it from a terminal window.

SUMMARY:

The twelve principles mentioned above might not seem novel, you might already be using some already in your engineering cycles. But if you are running a microservice architecture, its important that you take these principles seriously. You might not see the benefit right away, but it will become highly important when you are running multiple services across multiple environments.

Logging the right way

Logging is an important aspect of software development, one that is many a time ignored by developers.
it's important to have clean and helpful logs. They can come real handy when you have to debug an issue.

Below are a few important points to keep in mind while writing your logs

1. Using a good API

In my experience,I find the slf4j API better than the others.
Consider this:

in log4j:
logger.debug("Found employee : name: "+name+"designation: "+designation+" department: "+department);
This is difficult to read and has 3 string concatenation.

it can be rewritten with slf4j as
logger.debug("Found employee : name: {} designation:{} department: {}",name, designation, department);
Now, this is more readable, and avoids the string concatenation.

2. Watch what you are logging

Every time you log, watch what you are logging. Sometimes loggers can be the cause of many exceptions

Example 1: logger.debug("User Details :{}", user.getName());
This might throw a null pointer if the user is null

Example 2: logger.debug("User Details :{}", user);
This is an example where the hibernate object has been sent to the logger. This can give way to one or more of these issue :
Out of memory error, N+1 select problem, lazy initialization exception, logs storage completely used up

It' always a good idea only to log the ids of domain objects instead of the complete object. They are enough information for debugging.
Log no more than necessary in release/production, excessive logging can make you app slow.

The following logger format works best for me:
Who(UserName) , When (Timestamp), Where(Context, ServletOrPage,Database), What (Command), Result (Exception/Error)


3. Use categories

The severity values INFO, WARN, ERROR, and DEBUG, can be used to filter logs based on the severity.
The benefit that loggers have over System out is the logging categories and levels.
To read more about logging severity, visit
https://www.tutorialspoint.com/log4j/log4j_logging_levels.htm

Logging is an I/O intensive activity and can get in the way of the application's performance, particularly when logging is done in the traditional way that is, synchronously writing to a FileAppender in the context of application threads.
It can often contribute to heavy heap consumption and garbage collection.
hence, excessive logging can lead to performance issues. So be concise, and log relevant data that would help in debugging.

You can also use isDebugEnabled() to prevent filter the logs for production.

4. Log files locally
Use a file local to the server to write the loggers. It provides a local buffer and you aren't blocked if the network goes down.

5. Rotation policies
Log files can get huge. Maintain a rotation policy to roll logs periodically. You can also have a strategy to destroy or back up your log files after a period of time.

The power of Continuous Integration

What is Continuous Integration?

Continuous Integration is a methodology followed by development teams, where in they submit their codes periodically to a source control management system (SVN, ClearCase, Git etc.). Each time a new submission happens, a set of automated scripts run to build, deploy and test the build. This ensures that the current build is error free and a clean deployment.

There are many tools available for Continous Integration...TeamShip, GitLab, Codeship. The most commonly used one is Jenkins.
I will talk about the most commonly used CI tool, Jenkins.

Advantages

This is just the tip of the iceberg. There are many advantages that CI brings along:

1. Integrations are automated
The integration process can be completely authomated. Configure your Jenkins to poll the repository on a regular interval. Once the tool detects a check-in, it triggers a set of configured scripts to automatically buils and deploy the build to the server. Thsi saves time and also eliminates manual errors

2. Testing can be automated
Jenkins can be used to trigger a set of automated test cases each time a build is triggered. This ensures that the current build does not break any existing use cases.

3. Early detection of issues
CI can help find the issues before they can appear on the deployed application on the server, thus helping finding issues earlier and causing substancially less damage

4. Time Efficient
You save a lot of time lost due to debugging, manual integration and the deliveries can be done faster

5. Reduces manual work
Equates to saving time, money and less errors due to an authomated process

Features provided by Jenkins:

There are a numerous things that are provided by jenkins, among others

1. Addons: Jenkins can be easily extended by using over 400 plugins that are available.

2. Source Control: Jenkins can directly connect to all major version control systems and can poll periodically for changes. It can also check code back in to the version control

3. Job Configuration History Plugin: Tracks the changes to the job configurations, including who did it.

4. Automation: Schedule automatic builds based on cron expressions

5. Testing: Configure automated test cases to be triggerd on each build. This can trigger an automatic smoke test for each build.

6. Shell Scripting: Can add pre and post build operation by using basic shell scripts.

7. Notifications: Notifications can be configured to be sent to a set of people for the build status.

8. Access Control: The jenkins admin has an elaborate dashboard to provide multiple levels of access to various users.

9. Sequence the jobs: The jobs can be sequenced so that the successful completion of one can trigger another job.

10. Distributed builds: The master slave architecture supports distributed builds. You can build on one server and deploy on another. This reduces the load on the servers

11. Publish various reports: The jenkins plugins can be use to publish various reports in graphical or tabular form, e.g test coverage, code coverage and static code analysis reports.

How to make the best use of CI

Here are some best practices so that the CI features can be used to its best:

1. Maintain a single repository.

2. Make daily check-ins. It is a good practice to check in your code daily and not have any uncommitted code in your local system.

3. Automate the CI job to poll the version control periodically so that build are triggered every time a code is checked in. This way its easier to know which commit has failed the build.

4. Have automated test cases to run for each build to ensure nothing breaks due to a new build.

5. Maintain the unit tests, they can be triggered as a part of the build and verify the build.

6. Publish each build to server, so that the latest build is available for everyone to see.

7. Publish notifications to owners for failed builds, so that actions can be taken quickly.

8. Publish relevant reports. Eg: Test coverage, Code coverage, static code analysis reports etc. A good reports generates confidence to the stakeholders.

Summary

This requires commitment from individual team members to commit code frequently and ensure that they do not check in broken and untested code. So this would require to educate and empower the team members.
Adopting the Continuous Integration and Continuous Deployment will help catch the issues early in time and hence reduce risks of errors and ensure better quality.

GROUP BY MONTH, YEAR AND WEEK IN SQL

Say, You have a table from which you need to get the count of records, sum of amount and average amount on a weekly, monthly and yearly basis.

These are the queries:

Select
COUNT(id) as COUNT, SUM(AMOUNT) as SUM, AVG(AMOUNT) as AVG, DATEPART(WEEK, CREATED_DATE) as WEEKCOUNT
from TABLE
GROUP BY DATEPART(WEEK, CREATED_DATE);

Select
COUNT(id) as COUNT, SUM(AMOUNT) as SUM, AVG(AMOUNT) as AVG, MONTH(CREATED_DATE) as MONTHCOUNT
from TABLE
GROUP BY MONTH(CREATED_DATE);

Select
COUNT(id) as COUNT, SUM(AMOUNT) as SUM, AVG(AMOUNT) as AVG, YEAR(CREATED_DATE) as YEARCOUNT
from TABLE
GROUP BY YEAR(CREATED_DATE);

Conditional coloring of the list rows in Sencha Touch 2

I had a tough time doing this. Finally its up. You need to manipulate on the dom.
Just need to add the following listener on your list item.

listeners:{
                activate:function(list,index,item,record){
                var itemClass =Ext.get('id-of-your-list').dom.getElementsByClassName("x-list-item");
                    for (var i = 0; i < itemClass.length; i++) {                       
                        if(this.getStore().getData().items[i].raw.status ===1){ //condition on the record
                            itemClass[i].style.backgroundColor='#f3f3c7'; //change the color of the row
                        }else if(this.getStore().getData().items[i].raw.status ===2){
                            itemClass[i].style.backgroundColor='#c7f3c7'; //change the color of the row
                        }
                    }
                }
            }