Sunday, 29 March 2020

Nth Highest salary of the employee

Nth Highest salary of the employee, this is general question. I am giving answer my own style.

select * from employee e1 where n-1 = (select count(distinct(emp2.salary)) from employee 2 where
e2.salary > e1.salary);

Employee Table:
   id       salaray
    1          5
    2          4
    3          3

Nest query means query contain another query, nothing
e1 - for loop
e2 - nested for loop

1st pass:
                   e2.salary  >   e1.salary
                            5    > 5     NO
                            4    > 5     NO
                            3    > 5     NO


2nd Pass:
                  e2.salary  >   e1.salary
                            5    >  4    Yes          - This will match

Internally All sql queries using algorithms, I am not remember exactly either merge sort or bubble sor t  or selection sort they are using internally.

I want to highlight here is the bubble sort - it's matching above criteria.

Bubble Sort:
Main logic:
    for(int i=0; i < n; i++) {
         for(j=1; j < n-i; j++) {
                if(a[j-1] > a[j] ) {
                      temp = a[j-1];
                      a[j-1] = a[j];
                      a[j] = temp;
                }
          }
    }

The nested loop why it's using  j < n-i , I will give answer

2 3 4 5 1  -  Unsorted

1st Pass:  i = 0,  j < 5
  2 3 4 5 1    -   2 < 3 ok
  2 3 4 5 1    -   3 < 4 ok
  2 3 4 5 1    -   4 < 5 ok
  2 3 4 5 1    -   5 > 1 swap
  2 3 4 1 5

After 1st pass is over, you have filter highest element 5 in the last.

2nd Pass: i = 1; j < 4
     in the 2nd pass observer here j < 4 means in the 1st pass we had already filtered highest element 5.

  2 3 4 1 5    -   2 < 3 ok
  2 3 4 1 5    -   3 < 4 ok
  2 3 4 1 5    -   4 < 1 swap
  2 3 1 4 5    - 

     In the 2nd pass we had filtered 2nd highest element 4 in the last .


like this way it will perform.

Above nested sql query similar to the bubble sort.
 

Wrost case: O(n2), Average Case: O(n2).

Now we will see selection sort:
   1) First find the smallest element in the list and put it in the first position. Then find the second smallest element in the list and put it in the second position and so on..
 
       otherwise
   First find the highest element in the list and put it in the last position. Then find the second highest element in the list and put in the second last position and so on.

I am showing 2 examples

Approach Find Smallest Element:

selectionSort(int[] arr) {
    int i, j,minIndex,temp;
    int n = arr.length;
    for (i =0; i < n - 1; i++ ) {
           minIndex = i;
           minValue = arr[i]
           for( j = i+1; j < n ; j++ ) {
                 if( arr[j] < minValue ) {
                         minIndex = j;
                         minValue = arr[minIndex]
                  }
           }
            //once inner loop over, we will find the smallest element then swap
            if(minIndex ! = i ) {
                     temp = arr[i];
                     arr[i] = arr[minIndex];
                     arr[minIndex] = temp;
            }
}

In this approach 1st we fill find the smallest element put it in the first position.

Example:
Unsorted Array: 2 3 4 5 1

1st Pass; i = 0, minValue=2, minIndex = 0, j = 1

2 3 4 5 1  ;  j=1, 3 > 2  then minIndex = 0, minValue = 2
                   j=2 , 4 > 2 then minIndex = 0, minValue = 2
                   j=3 , 5 > 2 then minIndex = 0, minValue = 2
                   j=4,  1 > 2 then minIndex = 4, minValue = 1
Once first pass will be over check whether minIndex != i and then swap
i=0, minIndex=4
Now swap with a[i] with a[minIndex], first position "2" swap with "1"

In this after 1st pass we had filter minValue put it in the first position,

After the 1st pass:
1 3 4 5 2

for( j = i+1; j < n ; j++ ) {, 
Then for 2nd smallest element, here observer here starting with j = i+1 , means we already done 1st place, now in the 2nd pass identify i = 1, j = 3  , from 2nd to Nth

2nd Pass; i = 1, minValue=3, minIndex = 1, j = 2
1 3 4 5 2 ;   j=2,  4 > 3  then minIndex = 1, minValue = 3
                   j=3 , 5 > 3  then minIndex = 1, minValue = 3
                   j=4 , 2 > 3  then minIndex = 4, minValue = 2

After 2nd Pass identified 2nd minimum value, then put in 2nd position

After 2nd Pass

1 2 4 5 3

What I am trying to say means in selection sort also after every pass we will filter min value like this.

In the inner loops,
filter highest value put it in the last means   for(j=1; j < n-i; j++) 

filter smallest value in the first like selection then for( j = i+1; j < n ; j++ ) 

               





             



















                       

    
                 

Tuesday, 10 March 2020

LongURL to Short URL

Convertion of Long URL into Short Url ?
     Base 10 ( 0 - 9)
     Base 62 ( A-Z, a-z,0-9)
     Base 64 (A-Z,a-z,0-9,_,-)

Below example talks about Base 62 short url program.

Please refer:
https://www.geeksforgeeks.org/how-to-design-a-tiny-url-or-url-shortener/

Once you will know core logic, then you can apply this logic in Distributed System/ Monolothic System.

Refer:
https://medium.com/@narengowda/url-shortener-system-design-3db520939a1c

https://www.youtube.com/watch?v=JQDHz72OA3c

https://www.educative.io/courses/grokking-the-system-design-interview/m2ygV4E81AR?affiliate_id=5082902844932096&utm_source=google&utm_medium=cpc&utm_campaign=grokking-manual&gclid=Cj0KCQjw9ZzzBRCKARIsANwXaeKy4vuv8wgQqmbEDiH_W_8mt-33jBbLzbl_9pmy0wYLtiy7cZawaPYaAn6PEALw_wcB

Ramesh Comments: Main intension of this approach to save space, time and better performance.

Monday, 9 March 2020

Distributed System Design

This Distributed System Design, I will update based on my reading.
Today I had went through the twitter system design.

Twitter System Design:   https://imgur.com/6TFkVL2

 Twitter Technical Stack:
   1) Redis Cluster
   2) Apache Storm/Kafka Streams
   3) Search - EarlyBird
   4) http push websocket - million of mobiles
   5) Apache Zookeper - maintains nodes


 Refer: https://www.youtube.com/watch?v=wYk0xPP_P_8

Whatsup System Design

Whasup Techical Stack:
    1) Erlang, the functional programming language
    2) Webserver for load balancer - Yaws webserver
    3) Message Server - XMPP Server (XMPP protocol)
    4) Database - Mnesia / CouchDB Cluster
    5) Apache HttpServer for Media Handling



Refer: https://medium.com/codingurukul/whatsapp-engineering-inside-2-bdd1ec354748
           https://www.youtube.com/watch?v=L7LtmfFYjc4


Netflix System Design

Front End : React JS

Server: Netty Server (TCP Server), Apache EC2 Cluster(My sql), AWS ELB

Cache : EVCache (Memcached)

Streaming: Apache Kafka

Compute : Spark cluster, Apache Samza

Storage: AWS S3,Cassandra Cluster

Services: Microservices ( ZUUL, Hystrix etc), Amazon EMR ( Managed Hadoop Framework), Archer - MapReduce Style Platform, Elastic Search, Apache Chukwa (Apache Chukwa is an open source data collection system for monitoring large distributed systems) 

Service Orchestra: TITUS ( Container Managed Platform, like kubernets) AWS Application Auto Scaling feature ((Titus is a container management platform that provides scalable and reliable container execution and cloud-native integration with Amazon AWS))
   
Refer: https://medium.com/@narengowda/netflix-system-design-dbec30fede8d

Varnish  - Open source web cache

Below diagram talks about Replica Vs Sharding
Reference: Designing Distributed Systems By Brendan Burns ( Very Good Book which will cover main concepts)

Replica Service: Stateless Service, Assume Request A can answer by any of #1,#2,#3

Sharded Service: Stateful Service, Assume Request A can answer by #1


Now I am started to work on Kubernetes - Below is the best material to learn the Kubernetes
https://kodekloud.com/courses/675122/lectures/12039431
This author covered all the kubernetes topics very well.

To setup the minikube on window 10 home edition: https://www.assistanz.com/installing-minikube-on-windows-10-home-edition-using-virtualbox/

I had tried minikube is working with virtualbox 5.2.x only, for virtualbox 6.x I had got error. While installing the minikube in your local please refer the virutalbox.

Docker daemon is a process, it will come along with minkube, Hence in my local i am utilizing the minikube docker daemon.

Below is the command to check:
minikube docker-env

Result:
SET DOCKER_TLS_VERIFY=1
SET DOCKER_HOST=tcp://192.168.99.100:2376
SET DOCKER_CERT_PATH=C:\Users\rames\.minikube\certs
SET MINIKUBE_ACTIVE_DOCKERD=minikube


What is sticky session ?
2 Webservers - server1 and server2
Application Load Balancer - default time - 5 min

10:00 Client1   > ALB  > sticky session > server1
10:03 Client1   > ALB  > sticky session > server1
10:08 Client1   > ALB  > sticky session > server2 ( why ? because sticky session timeout 5 min means same client --> attach to server1 until 5 min, if any client1 request coming with in a 5 min, it will map to server1 means sticky session attached for particalr server).


What is Anti Corruption Layer ?
https://dev.to/asarnaout/the-anti-corruption-layer-pattern-pcd,  It is going to protect existing legacy system through combination of patterns those are facade, adapters and translators.

https://docs.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer












Sunday, 8 March 2020

Excel With Multi Threaded Way

Today I am talking one of my innovation Excel With Multi Thread Design.

Logical :

 Excel Workbook having sheets like s1,s2,s3 etc.
 Now assign every sheet one thread
 Now wait for join of all threads.

I am attaching my source code for download:
https://drive.google.com/open?id=1PFnLxCrwTjAizTKWpE_bXXUSqzq4IwA0



IBM WAS Context:
        A work manager is a thread pool created for Java Platform. In case if you are applying excel with multi thread design, using WAS admin console configure the workmanager. Then refer the workmanger for accessing the threads. 

Without IBM WAS workmanager applying Excel with Multithread design will give performance issue due to insufficient threads of jvm.

Refer: IBM Workmanager
https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.base.doc/asyncbns/concepts/casb_workmgr.html
      

Wednesday, 4 March 2020

Forward Proxy and Reverse Proxy (Nginix, Traefik)

Please refer : https://www.cloudflare.com/learning/cdn/glossary/reverse-proxy/

Explained very simple way.

Forward Proxy/Proxy Server: In any organization they will keep proxy server means all employees request will go through proxy server to connect internet.



I had explained about the proxy intension in my previous proxy blog.

Reverse proxy: All Webserver / application server will keep the proxy in front of them to restrict the access.

These proxy understanding important, In the next microservice architecture service mesh plays key role.

Microservice Netflix Zuul Service act as Proxy Server, means client validation etc..

Basics: Proxy Design Pattern ( Refer my blog).

Now on distributed system reverse proxy playing key role.

Reverse proxy implementors:1) Nginix
2) Traefik (Reverse Proxy - avoid most of configuration issue : https://www.infoq.com/presentations/traefik/?itm_source=infoq&itm_medium=videos_homepage&itm_campaign=videos_row2 ). 

I will update this topic while I am reading..






Sunday, 1 March 2020

LRU Cache

Now a days this is most favorite interview question .

1) First try to understand time complexity calculation
     for(int i=0; i < n; i++) {
     }
   
     Time complexity: Big 0(n), you need to go all n elements

2)  for(int i=0 ; i < n; i++ ) {
         for(int j=0;  j < n; j++ ) {
         }
    }

    Time Complexity: each elements n times, then n elements n times
    Big O(n2)
 
     My theory simple : how many nested loops that many square n elements

3) for(int i=0; i < n; i++ ) {
        for(int j=0; j < n; j++) {
          for(int k=0; k < n; k++) {
          }
        }
    }

    how many loops: 3 then Time Complexity: n * n * n = Big O(n3)


LRU Cache is tricky question. Here Time complexity is very important.

In the cache fixed num of slots are assume: 4

1) Initial we will fill the slots with numbers , Assume: 4, 3,2,1
2) Now any one search for 5, 5 is not in the list then it will evict the 1 slot, add the element at top means now 5,4,3,2

   Search for Number in the slot: O(1)
   Deletion of the slot : O(1)
   Insertion of the Slot : O(1)

In the above scenario - deletion at last, addition at front

So double linked list best it will take O(1), if we know the position then it will take O(1) for deletion and insertion

Search : O(1) - HashMap only have that feature

Now we need to combine both datastructures here.



Refer below for good explanation.
https://medium.com/@krishankantsinghal/my-first-blog-on-medium-583159139237

   

Thursday, 27 February 2020

PostgreSQL Streaming


Check Proxy will be working or not

export http_proxy="http://xxxxx.xx.xx.net:3128/"
export https_proxy="http://xxxxx.xx.xx.net:3128/"
export no_proxy=localhost,127.0.0.1,Host IP (LINUX HOST IP)
export DOCKER_HOST="tcp://127.0.0.1:2375"

After setting the exports, then check


  Getting 200 means proxy is working fine.



Debezium Architecture





Using Change Data Capture (CDC) is a must in any application these days.
No one wants to hear that the changes they made did not reflect in the analytics because the nightly or hourly sync job has not pulled or pushed the data. The common problem is that there are a raft amount of web applications which are OLTP (Online Transaction Processing) and are often backed by a relational database such as Oracle, PostgreSQL, MySQL etc.

Performing real-time data analytics on these database systems requires usage of big joins and aggregations which results in locks as these database systems are ACID complaint and provide good isolation levels.
These locks may be held for a long duration which could affect the performance of the application for the live users.

Solution - Change Data Capture Pattern:
-----------------------------------------------------
Thus, it makes sense to stream data into other teams of your organisation which could perform analytics on it using spark jobs, hive queries or whatever is your preferred framework for big data madness.
The following technologies will be used to accomplish capturing data change.
Apache Kafka — It will be used to create a messaging topic which will store the data changes happening in the database.
https://kafka.apache.org/
Kafka Connect — It is a tool used for scalable and reliable data streaming between Apache Kafka and other systems. It is used to define connectors which are capable of moving data from entire databases into and out of Kafka. The list of available connectors is available here.
Debezium — It is a tool used to utilise the best underlying mechanism provided by the database system to convert the WALs into a data stream. The data from the database is then streamed into Kafka using Kafka Connect API.
https://github.com/debezium/debezium


Capturing data from PostgreSQL into Apache Kafka topics.
Debezium uses logical decoding feature available in PostgreSQL to extract all persistent changes to the database in an easy to understand format which can be interpreted without detailed knowledge of the database’s internal state. More on logical decoding could be found here.
Once, the changed data is available to Debezium in an easy to understand format it uses Kafka Connect API to register itself as one of the connectors of a data source. Debezium performs checkpointing and only reads committed data from the transaction log.
Let us run an example
To run this example you will require docker.

Start a PostgreSQL instance
docker run --name postgres -p 5000:5432 debezium/postgres

Start a Zookeeper instance
docker run -it --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper

Start a Kafka instance
docker run -it --name kafka -p 9092:9092  --link zookeeper:zookeeper debezium/kafka

Start a Debezium instance
  One important point here:
                   Check in your docker instance echo $DOCKER_HOST, if it’s not there then
export DOCKER_HOST="tcp://127.0.0.1:2375"

checking cut command:
-----------------------------------------------------------
$(echo $DOCKER_HOST | cut -f3 -d'/' | cut -f1 -d':')
echo "tcp://0.0.0.0:2375" | cut -f3 -d'/' | cut -f1 -d':'

check cut command working or not.

docker run -it --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME=$(echo $DOCKER_HOST | cut -f3 -d'/' | cut -f1 -d':') --link zookeeper:zookeeper --link postgres:postgres --link kafka:kafka debezium/connect


Connect to PostgreSQL and create a database to monitor

docker exec -it postgres psql -U postgres
psql -h localhost -p 5000 -U postgres
CREATE DATABASE inventory;
CREATE TABLE dumb_table(id SERIAL PRIMARY KEY, name VARCHAR);

What we just did?



Create connector using Kafka Connect

url -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '
{
 "name": "inventory-connector",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max": "1",
 "database.hostname": "postgres",
 "database.port": "5432",
 "database.user": "postgres",
 "database.password": "postgres",
 "database.dbname" : "inventory",
 "database.server.name": "dbserver1","database.whitelist": "inventory","database.history.kafka.bootstrap.servers": "kafka:9092","database.history.kafka.topic": "schema-changes.inventory"
 }
}'

Response:
{"name":"inventory-connector",
 "config":{"connector.class":"io.debezium.connector.postgresql.PostgresConnector",
           "tasks.max":"1","database.hostname":"postgres","database.port":"5432","database.user":"postgres","database.password":"postgres","database.dbname":"inventory","database.server.name":"dbserver1","database.whitelist":"inventory","database.history.kafka.bootstrap.servers":"kafka:9092","database.history.kafka.topic":"schema-changes.inventory","name":"inventory-connector"},"tasks":[],"type":"source"}

Verify the Connector is created


curl -X GET -H "Accept:application/json" localhost:8083/connectors/inventory-connector

{"name":"inventory-connector",
 "config":{"connector.class":"io.debezium.connector.postgresql.PostgresConnector",
           "database.user":"postgres",
                           "database.dbname":"inventory",
                           "tasks.max":"1",
                           "database.hostname":"postgres","database.password":"postgres",
                           "database.history.kafka.bootstrap.servers":"kafka:9092",
                           "database.history.kafka.topic":"schema-changes.inventory",
                           "name":"inventory-connector","database.server.name":"dbserver1",
                           "database.whitelist":"inventory","database.port":"5432"},"tasks":[{"connector":"inventory-connector","task":0}],"type":"source"}


Start a Kafka Console consumer to watch changes

docker run -it --name watcher --rm --link zookeeper:zookeeper --link kafka:kafka debezium/kafka watch-topic -a -k dbserver1.public.dumb_table

Result

Reference: