rememberLauncherForActivityResult is basically used to get results from other activity components like launching a camera or document picker or contacts picker based on intents, we will use the same to launch our own custom intent in composable function and get results.

Google Auth Project Creation:

So the first thing that we need is to Configure a Google API Console project that will give us the OAuth 2.0 client ID. Once you have the OAuth 2.0 client ID paste into your local string XML file.

<string name="gcp_id">553313394578-6uuv2ohetikvrqkb6nekri7en4qbca74.example.comstring>

Add the Google auth SDK app dependencies:

implementation 'com.google.android.gms:play-services-auth:19.2.0'

Let’s write some more code:

Here we will first declare our Google sign-in client object in MainActivity

Now comes the important part where we will declare the rememberLauncherForActivityResult() in the composable function where signup UI is being added.

Assuming we have some button to launch the google sign-in Signup screen then we can do something like below. Call the newly created startForResult object to launch the intent inside button onClick lambda function.

And that’s it if you run it then you should be able to see the google sign in popup on click of the below button and get the results back in the rememberLauncherForActivityResult callback.

We have been doing this previously for UIToolkit but since we are adopting Jetpack Compose slowly so many challenges come up on how to do it with Jetpack compose the same task. I hope this small article helps in bridging a small gap in adopting Jetpack Compose. Next time we will learn something new about Jetpack Compose.

References:

https://developers.google.com/identity/sign-in/android/start-integrating

https://developer.android.com/jetpack/compose/libraries

The post Google Sign in with Jetpack Compose first appeared on Falabella.

• parallel testing,
• mock testing,
• data driven testing,
• API testing,
• UI automation,
• performance testing by integration with gatling

all of the above with one open source tool, then karate framework is the most suitable option.

Karate is based on java. It requires jdk 1.8 and maven 3.6.X as prerequisites.

Use Visual Studio code as the IDE with two plugins:

• Open default browser.
• Cucumber with full gherkin support.

To get started all you have to do is copy paste the below maven archtype in your terminal:

mvn archetype:generate \

-DarchetypeGroupId=com.intuit.karate \

-DarchetypeArtifactId=karate-archetype \

-DarchetypeVersion=1.1.0 \

-DgroupId=com.mycompany \

-DartifactId=myproject

Open the project in VS code and voila the framework structure is ready!

Karate supports both java and javascript but you don’t need a prior understanding on either languages to start coding. The test cases are extremely easy to develop as it uses Gherkin keywords with BDD syntax i.e; Given, And, When, Then.

This makes is convenient to be understood by all types of audience be it technical or non-technical. Knowledge transition and onboarding of new resources too is a simple task.

Framework Structure:

POM.xml: consists of the maven, junit and karate dependencies by default post karate installation.

Karate-config.js: here is where you can declare the constants like, auth, base url, env, reusable code that needs to be utilized by all feature files.

TestRunner.java: has the template for running all the feature files at once, you can also mention the tags here to run a selected group of scenarios.

src/test/java: this will hold the package, which can be further divided into two folders: features and resources

features →user.feature

resources → payloads → addUser.json patchUser.json updateUser.json

The feature file is where you have to do the development of scenarios.

The file name must have the extension “.feature”, say for example user.feature

Here is an example of a feature file with two scenarios.

You can go ahead and add multiple scenarios and multiple feature files as per your requirement.

Now you are good to do the test execution.

Command for test execution:

1. Use below command to run all feature files in a particular test environment:

mvn clean test -Dkarate.env=QA

OR

2. Use below command to run testcases tagged as “smoke” in a particular test environment

mvn clean test “-Dkarate.options= — tags @smoke” -Dkarate.env=QA

Report:

The target  karate-reports folder in the framework structure will store the summary report post test execution completion.

File name: karate-summary.html

You can right click on it and choose the “open in default browser” option and a colorful report will open stating the feature, scenarios , pass, fail, time taken to execute.

On clicking the feature it will expand to show all the scenarios in detail.

For failed scenario the report looks like this:

Built-in report is the best part about the framewok you don’t need any third party tool or plugin or any extra code for report generation.

There you go, framework for basic API testing is ready!

References:

https://github.com/karatelabs/karate

The post Karate Framework for API testing first appeared on Falabella.

Preview the data:

Most of us get comfortable only when we preview the data, before we start analyzing and building models . Like Andrej Karpathy says:

The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. This step is critical. I like to spend copious amount of time (measured in units of hours) scanning through thousands of examples, understanding their distribution and looking for patterns. Luckily, your brain is pretty good at this.

But when you are presented with a few gigabytes of data in a single file, it gets harder to open/preview the data using traditional tools like notepad++ or vim, which consumes the entire system RAM to load the files. One alternative is to use tools like less in UNIX, which helps us just get the preview of the data instead of loading the entire data. In Windows, there are options like EditPad Lite and Large Text File Viewer, although, I still prefer the using the more command from the powershell or even better, using all the benefits of Unix from WSL. Below if the data preview by running a less command on the train.csv from the CTR prediction dataset:

Once we preview the data visually, we at least know the basic traits of the file like the number of columns, column names (if present as a part of the header), delimiter, basic types of the fields (numeric, floating, string, etc.), presence of missing values (if you can capture it in the first few records). You could also compute the total records by wc -l command (word count)

Read Data

The next logical step would be to read this data into your system. Again, given the massive data size, it would be close to impossible to read the dataset into the system RAM.

Read Data line by line:

One option would be to read it line by line as per the example given below. Note: I have not added any preprocessing, but one may choose to perform any processing after the read step (at the comment). Also note that, the above code will read every variable as string. One may want to do necessary type conversions in the processing step.

This works well without running into any RAM constraints, and it takes about 70 seconds to read 5 .8 GB of data. This method will work in any data size, in any given system RAM.

Read Mini-Batch Data with Pandas:

While the above method works perfectly well on any small machine, we can see that it is pretty inefficient, as it processes line after line. This has two problems:

• We are not utilizing the complete RAM of the system and we can definitely do better on time
• Some of the data processing may require us to see more than 1 line of data (for example finding out the distribution of a variable)

This can be addressed by mini-batch read. One easiest way to read the data in minibatch would be to use chunksize in pandas.read_csv module as shown below.

You could also choose the columns that you wish to read, by specifying usecols option in read_csv, thus reducing the memory consumption even further.

Dask

Another alternative is to use dask, which utilizes clever distributed computing and lazy loading. This means that it uses multiple cores to read the data in parallel, and at times when when you run short of memory, it only loads the structure of the data and returns the actual data only when required. In the below example, you can see that it hardly takes a few micro-seconds to perform read_csv.

The below code summaries all the three approaches:

Numpy memmap

Speaking about lazy loading, if you happen to have only numeric data, then we can alternatively make use of numpy memmap, which only maps the addresses of the data on the disk. It actually fetches the data only when it is referenced with the indices when required. (https://stackoverflow.com/questions/43393821/how-to-concat-many-numpy-arrays)

ftrainY = np.memmap('data/trainY.npy',dtype=np.int,shape=(15727857,),mode='w+')
ftestY = np.memmap('data/testY.npy',dtype=np.int,shape=(3931965,),mode='w+')fread = np.memmap('data/trainY.npy',mode='r',shape=(15727857,))def chunker(seq, size):
    return (seq[pos:pos + size] for pos in range(0, len(seq), size))for batch in chunker(fread,500000):
    print(batch.mean())

Database based alternatives

Other ways to read such massive data would be to create a mini database like sqlite in your own system and use tools like sqlalchemy to read them with filters, etc. I’ll probably reserve that discussion for another post.

Building Models

Once the data is loaded, the next task would be to perform some variable reduction or some modelling. A lot of the readily available machine learning packages mostly expect the entire data to be loaded into the RAM for performing their ‘fit’ functions. Below I’ll illustrate some of the ways to ensure that the ‘fit’ happens irrespective of the data size.

Smarter Initialisers leads to faster convergence

A lot of algorithms are sensitive to the weight initialisers. A purely random weight initialisation would take a longer duration to converge as it will have to re-adjust its weights a lot more to the optimal points. Also poor initialisation may lead to exploding/vanishing gradients. There are smarter hacks to intelligently initialise your weights before beginning to learn. For example, if we are trying to perform a regression whose mean value is say — 250, then initialise your bias to be equal to 250. Also scale your input variables before fitting, so as to reduce the range of gradients while learning. For classification problems, initialise the logit bias such that your model predicts probability equal to the 1:0 ratios at initialisation. While clustering, determine the distribution of points by various features (Eg: get the 5th, 10th, 25th, 50th, 75th, 90th and 95th percentiles of each features) and initialise the centroids around these ranges.

Another alternative could be to fit a model on a very small sample and use these weights (in case of supervised models) OR centroids (in case of unsupervised models) as initialisers for the large dataset modelling.

Partial/Incremental fit vs full fit

A lot of the algorithms allow stochastic learning. That means, instead of learning from the entire data, they can learn if they were provided row-by-row data. This could even be extended to a mini-batch data (a small set of rows, instead of single row). That means, in a typical supervised learning setup, we start with some randomly assigned — learnable weights, and keep adjusting those weights as and when we encounter more data and labels. Below, I shall discuss some of the methods that uses these partial fit/incremental fit —

Data Scaling:

The data can be across different units of measurements (like percentages, large float numbers, small ranged integers, binary, etc.). In order to bring all these variables to a comparable scale, we perform scaling. The most popular API for scaling is StandardScaler or MinMaxScaler. StandardScaler computes the Mean/Standard Deviation of the variables and subtracts the mean from each row of the variable and divides it with the Standard Deviation, thus ensuring that distribution of the variable has a mean of 0 and standard deviation of 1. i.e. —

where μ is the mean of the variable and σ is the standard deviation of the variable.

Similarly, the MinMax Scaling is performed by taking the ratio of the difference of each row of the variable with that of the min value of the variable to the difference max value of the variable with that of each row of the variable. i.e. —

However, both the StandardScaler and the MinMax Scaler in the above equations will require the entire distribution to be made available at once (in order to compute the mean, standard deviation, max and min). But we still are stuck with the big data problem of not being able to load the entire data into the memory. In order to circumvent this, we employ incremental aggregation computation methods, which will compute the mean, standard deviation, min and max incrementally. Though this may not result in accurate values, these approximations usually hold well for the large data.

Incremental Mean:

• We know that a simple mean of a variable is expressed as the sum of the variable values divided by the number of observations

• By re ordering some of the terms as given below, we can express the mean of the variable upto the n-th observation as a function of the the mean upto the n-1 th observation

Thus, you can see that the mean for the n-th observation can be derived from the n-1 th observation. In other words, the mean of a variable can be calculated incrementally. Similarly, we can also show that the standard deviation too, can be calculated incrementally. And so is the min and max of a variable (keep a pseudo min/ max variable and keep reassigning that variable as and when you encounter the new min and new max in the variable observations).

Using the above incremental aggregation concept, scikit learn has enabled a partial fit function for both the scaler, which keeps learning the scale function everytime you pass a chunk of data to it. So the learning can be done chunkwise. Once the entire data is fit, you can use the scaler object for further transformations. Below code shows how it can be done —

Dimension Reduction:

• Principal Component Analysis is typically used to reduce the large dimension to smaller components, where each of these components are expressed as a linear function of all the underlying features, while ensuring that each of these linear components are orthogonal to each other. I would not delve more into the working of PCA as such, but would point out that there exists an IncrementalPCA API in the sklearn library. This takes in the data batch-by-batch and incrementally fits the Components, which turns out to be pretty useful while training a large data. Below is an example usage of IncrementalPCA on the same large data example. Note that we also scale the data before we fit the PCA.

• AutoEncoders: Any neural network based method will naturally allow stochastic OR minibatch learning. It requires us to use the fit method (earlier known as the fit_generator, now overloaded into fit), by passing in a data generator instead of the actual data. In the next section, I’ve shown a sample — generic data generator, that could be used for any data, and also built a sample autoencoder that can compress the dimensions, similar to a PCA. More discussions on how an AutoEncoder compares to a PCA is outside the scope of this post — so probably another post on this..
• Incremental Matrix Factorization: Matrix Factorization is a popular method that is used not only in dimension reduction, but also in recommender systems, generating user/item embeddings, etc. The latent dimension that is generated out of the matrix factorization an be used as the reduced set of dimensions that represents the matrix as shown in the figure below. The typical set up requires us to load the entire response matrix to the RAM and perform the matrix decomposition which gets very expensive as the data and the sparsity increases. Alternatively, one can use a stochastic learning framework to learn the responses using any common algorithm like Alternating Least Squares, etc. to Factorize the Matrix, thus, being able to decompose any large matrix, with smaller RAM size. One such application is explained well in Incremental SGD (ISGD) — J. Vinagre, et al., 2014. Below is a small snippet that can perform such incremental sgd in a simple response matrix setup

Clustering:

A lot of partial fit based algorithms are made available that can incrementally learn the homogeneous groups of observations and add them to the clusters. Some of the examples include MiniBatchKMeans and Birch.

MiniBatchKMeans randomly samples mini batches of observations during each iterations during the training. At the initial iterations, the centroids are created local to the sampled space, thus leading to major updates in the centroids as and when we sample different spaces over subsequent iterations. After enough number of iterations, this would converge to the true universal centroids. In order to decrease the major updates to the centroid at each iterations, owing to the random samples, one hack could be to update the centroids by taking an incremental average (there we go again!!) of the current centroid position with respect to all the previous mini batches.

Birch on the other hand, creates trees called as Clustering Feature Tree, which can be treated as a lossy compression. The leaf nodes can then be treated as centroids. For more details, I recommend this blog by Cory Maklin which is quite intuitive to understand.

The codes for partial_fit for MiniBatchKMeans and Birch are well documented in the scikit-learn site.

Classification/Prediction:

Supervised learning can be performed either using Stochastic Gradient Descent’s partial fit or by using the incremental least squares (Code given below). The idea is similar, where we read mini batch observations and try adjusting the weights stochastically. Scikit-learn has a bunch of algorithms like SGDRegressor, SGDClassification, MultiNomialNB and BernoulliNB (Naive Bayes), PassiveAgressiveClassifier, Perceptron.

One can also tweak the LeastSquare algorithm to incrementally update the co-variance matrix, that can enable us to partially fit the data. Below is a code that I found on stackexchange, which does this —

import pandas as pd
	import numpy as np
	file_path = ‘../data/train.csv’
	chunksize = 1024
	X_vars = [‘hour’,’banner_pos’,’C1′,’device_type’,’device_conn_type’,’C14′,’C15′,’C16′,’C17′,’C18′,’C19′,’C20′,’C21′]
	y_var = ‘click’
	meanX = np.zeros((chunksize,len(X_vars)))
	meanY = np.zeros(chunksize)
	varX = 0
	covXY = 0
	meanXY = 0
	varY = 0
	alpha=0.001
	betas = np.zeros(len(X_vars))
	df = pd.read_csv(file_path,chunksize=chunksize)
	c = 0
	for chunk in df:
	X = chunk.loc[:,X_vars]
	y = chunk.loc[:,y_var]
	dx = X – meanX
	dy = y – meanY
	dxy = (X*y) – meanXY
	varX += ((1-alpha)*dx*dx – varX)*alpha
	varY += ((1-alpha)*dy*dy – varY)*alpha
	covXY += ((1-alpha)*dx*dy – covXY)*alpha
	meanX += dx * alpha
	meanY += dy * alpha
	meanXY += dxy * alpha
	betas = covXY/varX
	bias = meanY – np.dot(betas,meanX)

Tensorflow too, supports fitgenerator (now overloaded with fit function itself), where once can write a generator method to push the data in a streaming fashion for mini-batch stochastic learning. Tensorflow has a ready implemented image generator. Below is an implementation of a sample custom fitgenerator.

Conclusion

In summary, do not get bogged down by the size of the data. Bring on all the data in the world and with the right set of tools (both mathematical and computational) and the Attitude to solve them, we should be able to take on any monster to tackle.

Let me know your comments and thoughts.

Originally published on Medium

The post Build Models on Massively Big Data using Continuous/Online Learning first appeared on Falabella.

To overcome these challenges, we could go for cascading approaches which is one of the good approaches.

Below approaches can be followed in a cascading approach manner.

1. Title Similarity detection of different products

2. Image Similarity detection of different products

3. Attribute extraction/detection of different products

4. Price comparison of different products

Once the similarity is detected using the above approaches get the probability if the products are matching or not matching.

TITLE SIMILARITY MODEL:

IMAGE SIMILARITY MODEL:

DESCRIPTION/ATTRIBUTE EXTRACTION MODEL:

The above architecture can be used to compare the titles description but if we need to compare the features inside the product titles in different websites or link, we should go for NLP techniques to extract (using OCR) and then compare using Regex or NLP techniques. Price is also one of the feature that can be extracted and compared using the same technique.

Originally published on Medium

The post Retail Product comparison Architecture using different AI/ML/NLP techniques first appeared on Falabella.

Reinforcement learning — Step by Step Implementation using SARSA

In this tutorial, I have given the step by step implementation of Reinforcement Learning (RL) using SARSA algorithm. Before jumping on to coding and functions, I have briefly explained the subject to easily understand the code and the steps that have been followed one after the other. If this is your first time learning about RL-SARSA, I hope you will benefit from my tutorial below. I will also try to enhance this blog with the latest information as much as I can.

Take an example of a driverless scooter taxi (which performs the work of food delivery person) taking the food parcel from one place to another place.

Image:

Before proceeding further on implementing RL, we should know the following:

The main processes of RL are:

Observe, Decide, Act, receive, learn and Iterate

Observe means observing the environment of the agent

Decide means decide as per the observation

Acting means taking action on the decision

Receive means receive rewards or get penalized as per the action

Learn means to learn from previous actions and improve

Iterate means to repeat the entire process until success or the goal has been reached

The main components of RL are

Environment, Agent, Rewards, Goals and Actions, State, Policy

An agent isa learner and the decision-maker for eg: Taxi

The environment is the place where the agent learns and decides what actions to perform.

Action is a set of actions that the agent can perform.

State is the state of the agent in the environment.

A reward is nothing but a scalar value. For each action selected by the agent, the environment provides a reward. It can be a positive or negative reward. It can be instant or long-term reward

We need the above components to implement RL and to build Q table/Optimal policy

With respect above example with image, let’s arrive at these components and build Q table/Optimal Policy

States:

In the above image, think that it is an area divided to 5*5 grid which means it has 25 possible states. Current location of the scooter or bike taxi is (2,3).

This is just for an example. In a real-time scenario, the no of possible states might be in millions also 🙂

Here Food — states can be R, G, Y, B and inside scooter also and hence 5 states.

Locations where we can pick the food /delivering the food are 4

Hence Total number of possible states =

25 possible states * 5 Food states * 4 Locations for pick the food/deliver the food

= 500 possible states

Actions:

Actions could be the scooter moving up, down, right, left, picking the food, delivering the food. Hence totally 6 actions

Rewards:

When it comes to the rewards part, the scooter will receive a reward whenever it takes action.

It can be positive or negative. Rewards will be defined by the programmer and let’s define the

rewards for this example as below:

Reward to pick up the food from the right place: 10

Reward to deliver the food in the right place: 20

wrong moves = -1. This is called penalization when it does the wrong action. E.g.: Scooter has gone to the dead-end of the road, it will get a reward of -1.

Gym Package:

Gym is a toolkit which helps to develop and compare reinforcement learning algorithms. The gym library is a collection of test problems — environments — that anyone can use to test your reinforcement learning algorithms. These environments in the gym have a shared interface, which allows you to write general algorithms.

We have environments like FrozenLake-V0, Atari, 2D and 3D Robots, Taxi etc

The other env names can be found in the below link:

https://github.com/openai/gym/blob/master/gym/envs/

In OpenAI’s gym package: following functions can be used for implementing RL

env.reset will return a random initial state and also resets the environment. We can also reset to a particular state.

env.step(action)will increase the time step in the environment.

env.render will be helpful in the visualization of the environment with agent location.

Observation indicates the observation of the environment

A reward is a reward achieved from the action done previously. It can be positive or negative

Done will tell if we have reached the goal or not. It will Be False if not reached else True.

Now let’s get in to step by step implementation:

Step1: #Import the following libraries

#Pickle file is mainly to store our Q table and optimal policy #information

Step2: #Build /Create the environment

(For now, we will use the existing environment in the gym )

To check for other environments in the gym, you can sue the following code

Step3: Initialize all the parameters along with the Q table as below:

#Q table initialization as below:

Initially, the values of the Q-table are initialized to 0. An action is chosen for a state. As we move, Q value is increased for the state-action whenever that action gives a good reward for the next state. If the action does not give a good reward for the next state, it is decreased.

Step 4:

PART-A:

Now proceed to define the Epsilon greedy method and SARSA algorithm to build Q table

We have to build a table called Q-table which will have dimensions SxA where S is the number of states and A is the number of actions.

For Every S (State) there are A’s (actions), the likeliness of selecting or choosing a particular action depends on the values in Q table. Those values in the Q table are also known as State-Action values.

SARSA depends on the current state, current action, reward obtained, next state and next action. SARSA stands for State Action Reward State Action which symbolizes the tuple (s, a, r, s’, a’). SARSA is an On Policy, a model-free method which uses the action performed by the current policy to learn the Q-value

#Write the below function to Choose the action based on Epsilon greedy method

In the above function if the random number generated is less than 0.1, we can go for Exploration else we can go for Exploitation (Q learning).

Exploration means we should try to get more details about the environment.

Exploitation means we have to aim for maximizing the reward by making use of the information which is already found.

#Use below SARSA formula to learn Q value and update Q table:

In the above formula, we have the current station and action, Next state and next action. We also have alpha which is the learning rate and gamma as the discount rate.

PART-B:

#Write a Function as below to create a Q table. This will include SARSA formula

Step 5: Do the training using epsilon greedy method and SARSA algorithm.

Here the hyperparameters are your total_no_episodes.

Step 6: #Evaluate your SARSA model by using the reward and total number of episodes information

Step 7: #Get the Q table and visualize the matrix

Sample Q with dimensions (SxA -> 500*6) can be as below:

### Note the above Q table is just a sample

Step 8: Randomly select the state and see how many steps it has taken to reach the goal using the Q table that is already created with the above steps

# sample visualizations of first 2 steps and last few outputs are as below (performed with env.render())

…… …. Other steps with visualization will be displayed …

In the above outputs, the highlighted marks determine the current position of the scooter taxi in the environment while the direction given in brackets gives the direction of movement that the agent (scooter) will make next.

Finally, the food is delivered/ dropped in the right place after 14 steps in the above case

Step 9: Load the Q table in a pickle file for later use and unload the same

Referred links:

https://github.com/openai/gym/

https://www.geeksforgeeks.org/

Originally published on Medium

The post Reinforcement learning – Implementation using SARSA first appeared on Falabella.

One such document is the test strategy document which is always confused with the test plan document.

Let’s clear out this confusion first.

Test strategy document is a one-time layout stating the testing approach for all the releases while

Test plan document needs to be prepared for every release.

Now let us look at each document individually to understand the basic attributes required.

Test strategy document

Template of the test strategy can have the below:

· document name, version, details of author, reviewer, approver,

· technology stack,

· environment,

· types of testing,

· tools used for each type of testing,

· resource responsibilities,

· risks and mitigations

Let’s get into the details of each parameter. The name of the document should call out the project name.

Format example:TestStrategy_ProjectName_version.

Provide the below information for audit purpose:

It is important to know the software and hardware specifications of the application in order to predict the best tools that can be used for manual, automation and performance tests. Define the technology used and all the tools involved clearly.

Specify the environments that will be used in each stage of testing. For example, it is a common practise to have one lower environment for testing both functional and regression and another production like environment to perform a quick test of sanity along with the priority 1 scenarios identified from the current release scope.

Finalise the types of testing that will be carried out. This will differ from application to application. For example, an application driven by APIs and events will require, manual exploratory testing, automation testing along with basic security and performance testing.

Once the types of testing are finalised it’s time to identity the tools that will be used for each of them. For the above scenario, it is suitable to use postman for manual exploratory testing, rest-assured with testNG for automation testing and Gatling for performance testing.

Pick your tools depending on the make of your application.

Now identify the number of resources that will be required for each type of testing. State their responsibilities. Some projects have different teams for manual, automation and performance testing while others have same team for manual + automation and different for performance. Decide this structure and number based on the billing involved, skillset and timelines.

At last, foresee any anticipated risks and come up with their mitigation plans. There is never a plan without risks. It is the way we handle them that describes the success.

The test strategy once reviewed and approved will remain same throughout the project tenure.

Test plan document

On the other hand, a test plan has to be designed from the scratch for every release of the project. Template of the test plan must state clearly the below:

· document name, version, details of author, reviewer, approver,

· in scope functionalities,

· out of scope functionalities,

· risks and mitigations,

· entry criteria,

· exit criteria.

How to fill up each of these sections?

Name of the document must be easy and focussed on identifying the specific release. Format example: TestPlan_ReleaseName_version.

It is a good practise to record the below information for audit purpose:

The next page must be about the user stories committed for the release. Place them in the “In scope” section. Listing the user stories with their description will suffice.

In case due to any reason a user story was moved out of the release mention it in the “Out of scope” section.

Every testing phase will have a set of defined activities. Timelines section is very important to know the start date, end date of every activity to monitor the deviation if any.

In case of deviation between the estimated start date and the actual start date a risk must be raised accordingly.

Be proactive in identify any risks anticipated for testing the release. Types of risks could be-

· a resource crunch,

· test data unavailability,

· environment instability,

· slippage of code delivery date, etc.

Any risk mentioned should also come with a mitigation which will be the backup plan for solving the risk. For example, in a resource crunch situation, either hire a new resource or borrow one from another team of the same project account or in worse case ask if the existing resources can extend their shift or log in during the weekends (which is a bad idea ).

If test data is not available in lower environment, think about refreshing the database of production with that of lower environment.

Work with the infra team to make the environment stable.

Be ready with a priority 1 and priority 2 suite of test cases if the code delivery date is slipped so that with the available time only the priority 1 test cases can be finished successfully.

Entry Criteria section must be strictly followed as an acceptance criteria to begin testing.

· Stable code without severity 1 and 2 defects,

· Demo of the functionality and code walkthrough by developer,

· presence of integration and unit tests at code level, etc. are a few to name.

Exit Criteria is very essential to understand if the code is of good quality and can be promoted to the higher levels. For example:

· successful execution of all designed test cases,

· absence of any severity 1 and 2 defects,

· severity 3 bugs if present must have a workaround,

· defects that might not hamper the existing code and can be taken in the next or subsequent releases must be deferred after review with stake holders,

· demo of a happy path to end user.

Now that we are clear with the difference between both the documents, it is left to respective teams to decide and incorporate this documentation process based on time/requirement constraints.

Originally published on Medium

The post Test strategy and Test plan first appeared on Falabella.

One thought always comes to our mind that “What is Spark ? and why Industries are running behind it ?, also why Spark is ruling data space ?”. So hang tight and let me start with very funny example . Think of a beehive. You have a single queen and hundreds or thousands of worker bees. They have very distinct roles. The queen is largely responsible for the ‘brains’ of the entire operation, conducting the orchestra of worker bees who are fulfilling the hundreds of tasks that need to be accomplished. The worker bees are the executors, putting in the work required to accomplish those tasks.

What do Bees have to do with Apache Spark?
I feel my beehive example confused you more but be patient and let me help you to clear your confusion. They’ll come in handy soon. The formal definition of Apache Spark is that it is a general-purpose distributed data processing engine. It is also known as a cluster computing framework for large scale data processing.

Spark is built to handle extremely large scale data. The sheer amount of data being loaded into the spark application is enough to overwhelm almost any computer. To handle that, Spark utilizes multiple computers (called a cluster) to process the tasks required for that job and work together to produce the desired output.

This is where the bee analogy comes in. Let’s start with a diagram.

You will be laughing with this comparison. But let me explain you this diagram:

Spark has two main components, Spark driver and spark executors same as beeline queen and worker bees.

Spark Driver: The Queen Bee of the operation. The Spark Driver is responsible for generating the Spark Context. The Spark Context is extremely important since it is the entryway into all of Spark’s functionality. Using the Cluster Resource Manager (typically YARN, Mesos, or Standalone), the Driver will access and divide work between the cluster of Spark Executors (worker nodes). The Spark Driver is where the main method is run, meaning that any written program will first interact with the driver before being sent in the form of tasks to the worker nodes.

Spark Executors: The worker bees. The executors are responsible for completing the tasks assigned to them by the driver with the help of the Cluster Resource Manager. As they perform the tasks instructed to them, they will store the results in memory, referred to as a cache. If any one of these nodes crashes, the task assigned to that executor will be reassigned to another node to complete the task. Every node can have up to one executor per core. Results are then returned to the Spark Driver upon completion.

Apache Spark also takes advantage of in-memory caching for fast analytic queries for any data size. An in-memory cache is designed to store data in RAM and not on disk. You can use languages like: Scala, Python, R, and SQL to leverage Apache Spark.

Note: Spark is not a programming language. It is a general-purpose distributed data processing engine. Spark is written in Scala, a functional programming language. Spark, fortunately has a great Python integration called PySpark (which is what I use mostly) — it lets me interface with the framework in any which way I want.

How Apache Spark works ?

Now, you will be curious to know that how it works. So, let’s take a
further deep dive and understand spark architecture:

The first block you see is the driver program. Once you do a Spark submit, a driver program is launched and this requests for resources to the cluster manager and at the same time the main program of the user function of the user processing program is initiated by the driver program.

Based on that, the execution logic is processed and parallelly Spark context is also created. Using the Spark context, the different transformations and actions are processed. So, till the time the action is not encountered, all the transformations will go into the Spark context in the form of DAG that will create RDD lineage.

Once the action is called job is created. Job is the collection of different task stages. Once these tasks are created, they are launched by the cluster manager on the worker nodes and this is done with the help of a class called task scheduler.

The conversion of RDD lineage into tasks is done by the DAG scheduler. Here DAG is created based on the different transformations in the program and once the action is called these are split into different stages of tasks and submitted to the task scheduler as tasks become ready.

Then these are launched on the different executors in the worker node through the cluster manager. The entire resource allocation and the tracking of the jobs and tasks are performed by the cluster manager.

As soon as you do a Spark submit, your user program and other configuration mentioned are copied onto all the available nodes in the cluster. So that the program becomes the local read on all the worker nodes. Hence, the parallel executors running on the different worker nodes do not have to do any kind of network routing. This is how theentire execution of the Spark job happens.

Differences b/w traditional Hadoop map-reduce and Spark: For those of you who have traditionally worked on map-reduce or Hadoop, you might wonder why Spark? Spark has a big margin of advantage starting from speed to easy of writing. Some of the stark differences are:

Why its ruling ?

There are many reasons for its ruling but I would like to list out few of them. First let me tell you that “What makes spark so special?” So, this is a simple three-fold answer. 1. SPEED, 2.EASE OF USE, 3. A UNIFIED ENGINE.

But let’s deep dive into few reach features.

1. Fast processing:– The most important feature of Apache Spark that has made the big data world choose this technology over others is its speed. Big data is characterized by volume, variety, velocity, and veracity which needs to be processed at a higher speed. Spark contains Resilient Distributed Dataset (RDD) which saves time in reading and writing operations, allowing it to run almost ten to one hundred times faster than Hadoop.
2. Flexibility:- Apache Spark supports multiple languages and allows the developers to write applications in Java, Scala, R, or Python.
3. In-memory computing:- Spark stores the data in the RAM of servers which allows quick access and in turn accelerates the speed of analytics.
4. Real-time processing:- Spark is able to process real-time streaming data. Unlike MapReduce which processes only stored data, Spark is able to process real-time data and is, therefore, able to produce instant outcomes.
5. Better analytics:- In contrast to MapReduce that includes Map and Reduce functions, Spark includes much more than that. Apache Spark consists of a rich set of SQL queries, machine learning algorithms, complex analytics, etc. With all these functionalities, analytics can be performed in a better fashion with the help of Spark.

Scope: There are almost all the area and everyone can use spark in data space. Spark is a one-stop solution for many Big data-related problems. Some of the common use cases for Spark are:
1. Batch Processing (ETL)
2. Real time, well almost real time.
3. ML and Deep Learning

Data scientists use Apache Spark to perform advanced data analytics. Python brings an extensive set of advanced analytical functions that can be performed on data in Spark. Python is one of the more popular languages of the data science community and is also supported by Spark via a toolset called pySpark.

Data engineers are data designers or builders. They usually assist data scientists and application developers in the data curation journey. They develop the architecture for the organization based on use cases and needs.

Application developers can build solutions using Apache Spark. These applications are generally for analytical and business intelligence purposes. Spark is great for data analysis style applications and not for transaction processing applications.

BOTTOM LINE: Apache Spark was built in 2009 and made resilient by 2012 because MapReduce was slow and complicated, and people wanted something faster and easier. Apache Spark is fast and is far simpler to program than MapReduce. Imagine shoving a bunch of data into computer memory and being able to read it, process it, or do something rapidly. That is Apache Spark.In layman’s terms MapReduce was slow! SLOW + BIG DATA = NO JOY, thus we get Spark.

Apache Spark requires a decent amount of technical knowledge to make work. The average business person will require lots of help to get running on Apache Spark (being very generous). It is for programmers, data scientists, and highly technical unicorns.

CONCLUSION: Spark is an incredible technology and this article only touches the surface. I didn’t go into RDDs and other aspects but maybe in a future article I will. Hope to see you next time when I talk about installing and using Apache Spark in Kubernetes!

Thanks for reading, cheers!

Originally published on Medium

The post Why Spark ruling Data Space !! first appeared on Falabella.