When to Use after_commit in Rails ?

Active Record Callbacks helps us to execute code at certain stages of object life-cycle.

Let’s take a scenario, where whenever a user is created, you want to send mail to the user. So, in order to achieve this we will use after_save callback.


# app/models/user.rb

class User < ApplicationRecord

  after_save :send_email_notifications

  def send_email_notifications

    SendEmailNotifications.perform_later(self.id) # This will send email notifications in the background

  end

end

Now, this looks perfectly fine right ??

But, there is one issue with this code.

Race Condition
In the above scenario, after_save callback will be executed when the user record is saved to database. But what if commit transaction is not yet completed ?

In this case it will still send the email notification and it will raise NotFound Error while performing SendEmailNotification job.

 

So, how to overcome this?

HOW TO SOLVE THIS ?

In order to solve this, we need to use after_commit callback in place of after_save.

after_commit callback will be performed only when the actual commit transaction happens to database.

 
# app/models/user.rb 

class User < ApplicationRecord 
  after_commit :send_email_notifications, on: :create
  
  def send_email_notifications
    SendEmailNotifications.perform_later(self.id) # This will send email notifications in the background 
  end 
end 

So, we should always think before using callbacks, which one of them will satisfy our requirements and then use it.

Advertisements

Command Injection in Rails

Command Injection is a type of injection where attackers gains access to your system through your application. It occurs when a system command includes user manipulatable values.

Here are the common command lines which we generally use in rails application:

  • `….`
  • system()
  • %x[….]
  • exec()

Now, this is what command injection looks like:


path = "#{Rails.root}/public/#{params[:name]}"

`mkdir -p #{path}`

`ls #{params[:file]}`

Here, as the params data is provided by user, so it can be used by attacker to manipulate our system command and execute it.

In the above case, the attacker can just pass params[:file] value as  ; cat ./config/database.yml  and thats it. Just by doing this they have gained access to view your database credentials which is obviously sensitive.

So, how to prevent this from happening?

In order to prevent it from attack like command injection, we need to break our command into separate strings like this:

 

path = "#{Rails.root}/public/#{params[:name]}"

system("mkdir", "-p", path)

system("ls", "params[:file]")

Hope you liked it, for more such information stay tuned 🙂

Expression Indexes and Operator Classes in PostgreSQL

In this blog we are going to explore expression indexes and Operator Classes in PostgreSQL and its support in Rails. But before diving deep into it, we should be aware about terminology like Indexes.

So, moving on what is Expression Indexes ??

In order to understand that, let’s take a scenario, suppose there is a movies booking application and on one page we want to filter movies on the basis of movie name. So, what would be the query for the same ??

It will be following in:

PostgreSQL:

  SELECT * FROM movies WHERE lower(name) = 'name';

 

Rails:

  Movie.where("lower(name) = ?", name.downcase)

 

Okay, so everything looks perfect, but what if the number of movies increases?

In that case this query will take time. So, in order to optimise it we need to do indexing for this column name.  But in this case normal indexing will not work because here we are using expression lower(name) in the where clause. 

For such cases, PostgreSQL helps us by perform Indexing On Expressions

PostgreSQL(supported above  or equal to 9.4 versions) :

  CREATE UNIQUE INDEX index_movies_on_name_unique ON movies (lower(name));

 

Rails(supported Rails 5) :

  add_index :movies, "lower(name)",
            name: "index_movies_on_name_unique",
            unique: true

So, our problem is solved now right ?

Yes, regarding the above scenario, expression indexing solves our issue.

But what if on that page, we need to perform partial search on the basis of movie’s name ?

In this case, our query will look like this:

Rails:

  Movie.where("lower(name) like ?", "%#{name.downcase}%")

Here, in PostgreSQL, the indexing which we did earlier using expression indexing will not be used, instead it will go for sequential searching.

Therefore, for this we need to remove expression indexing and add operator class to the previous index, so that during searching it will use this one instead of doing sequential search

PostgreSQL(supported above  or equal to 9.4 versions) :

 CREATE UNIQUE INDEX index_movies_on_name_unique ON movies (lower(name) varchar_pattern_ops);
Rails(supported Rails 5) :
 
 def change
  remove_index :movies, name: :index_movies_on_name_unique
  add_index :movies,  'lower(name) varchar_pattern_ops',
                        name: "index_movies_on_name_unique",
                        unique: true
 end 

Hope you liked it, for more such information stay tuned …. 🙂

What is this Rack?

We as developers use frameworks like Rails, and while using it everything seems like magic to us because of abstraction. But have you ever thought of how the internal Request/Response cycle works in these frameworks Or who is the one driving it internally ?

 

HOW?

 

So, to solve everyone curiosity, RACK is the one which is responsible for handling all these things.

But what is this RACK??

What is that?

 

Rack is the middleware which helps application server to run your rails app. You can think of it as a common language that both your app server and rails application understands. So, our app server interacts to our rails application using rack(translator).

 

RACK INTERFACE

Rack defines a simple interface and it must have these three properties:

  1. It must respond to Call method
  2. Call method takes a single argument, termed as env/environment. This env contains all the data about the request
  3. Call method returns an array of three elements which is [status, headers, body] of the response

 

Lets understand this by writing a simple rack complaint application, which returns the text “Hi, I am RACK” 🙂

 
require 'rack' 
require 'thin' 

class RackApp 
 def call(env) 
  [ 200, { "Content-Type" => "text/plain" }, ["Hi, I am Rack"] ] 
 end 
end 

Rack::Handler::Thin.run RackApp.new 

In the above example we can see that RackApp is a class which has one instance method termed as
call which takes env as a argument and returns an array containing three elements:

  • Status Code: 200(HTTP “OK” status)
  • Header: Content-Type(text/plain)
  • Body: [“Hi, I am Rack”]

Here one thing to note down is that Body of the response must respond to each, thats why normal string is enclosed in an array and then returned. But in case of some type of IO object this enclosing inside array is not required.

 

Hope you liked it. In the next blog we will be covering little complex designing using Rack interface. So for more such information stay tuned …. 🙂

 

Partial Indexing In PostgreSQL

In this blog, we are going to learn about Partial unique indexing in PostgreSQL and Rails. But before that, I think you all should know about PostgreSQL Duplicate Null Values in Unique Column which I had written in my previous blog.

So, first of all what is Partial Indexing ????

According to the PostgreSQL documentation,

partial index is an index built over a subset of a table; the subset is defined by a conditional expression (called the predicate of the partial index). The index contains entries for only those table rows that satisfy the predicate.

In simple terms, it is process of adding unique index only on particular portion of database records, and that portion is defined by some condition.

So, in order to understand this, let’s take a scenario, where we have users table and we have email column, whose value must be unique. It means no two users can have same email, and we are deleting these users using soft deletion mechanism. Soft Deletion means deleting user from database will not remove it permanently from database, but will set deleted_at column value(current_timestamp). This column is used for categorizing deleted records from persisted ones.

Here, we have uniqueness constraint applied on email column, so if we delete one user with a email say “test@test.gmail”. And later if somehow we want to create user with same email, it will throw error, when we are using our conventional unique constraint. Because even though we have deleted user, it still persists in our database as it is soft deleted.

What is the solution then????

Okay at first, you all will think that creating unique index on the combination of these two columns(email, deleted_at) will solve the problem here.

  CREATE UNIQUE INDEX users_email_deleted_at_idx
  ON users (email, deleted_at);

But dear folks, this will not work.

WHY NOT????

Do you all remember this….?

PostgreSQL standard do not consider two null values as same, which I had already discussed in my previous blog.

  -> user1 = { "email": "test@test.com", deleted_at: null }
  -> user2 = { "email": "test@test.com", deleted_at: null }

Because of this, these user1 and user2 will be treated as different entity.

So, is there any solution for this ????

Yes, we have i.e our saviour “PARTIAL INDEX”

But, how will we implement partial index ?

In order to achieve this, we will add conditional index using “where” clause like mentioned below:

  CREATE UNIQUE INDEX users_email_deleted_at_null_idx
  ON users (email)
  WHERE deleted_at IS NULL;

Yayyy, now this will solve our all issues 🙂

BONUS: Also, we have active record helper for creating partial indexing in rails i.e

  add_index :users, :email, unique: true, where: "deleted_at is null"

Hope you liked my blog. For more information like this stay tuned ….. 🙂

EXPLORING REDIS AND IT’S DATATYPES

Today Performance is what that comes first, when we developers try to develop web services. One of the issue is that, when a web service tries to interact with database, in order to get the result it may take time depending on the number of records.

Prerequisites

For this blog, I am assuming that you have knowledge about Rails and basic idea about Redis.

Getting Started

Lets’s imagine we are building a back-end for the online movie app. Customers will use this app to view all the movies, their details, resulting in huge load on Database. So what if we could reduce the load on the database by caching the movies data. But for caching what should we use ?

There comes REDIS to our rescue.

Redis

Redis is the key-value store, which we can use for CACHING to speed things up and improve our performance.

But Redis is not just a plain key-value store, it is data structures server, means it not just limited to support strings as value, but also more complex data structures, such as Hashes, Lists, Sets, Sorted Sets. For detailed information refer this.

Strings

Strings are the most basis data type that we use for caching in Redis. They are binary safe and easy to use. So we mostly go for them.

But in our scenario Strings DataType was not enough as I have to store the whole list of movies and their respective details in Redis.  Strings work well, but it stores the whole list in the string format as value. So, before sending the data , I have to parse them in JSON Format,  such that they can be used by the views in order to present it to the User. But what if the data is huge, parsing strings to JSON or any other required format will be time consuming. So, string is not which can be used in our case.

By reading these memory optimization blog and documentation, I found that there is other Datatype that Redis supports, which can be helpful i.e, Hashes.

Hashes

Hashes are the perfect data structure to represent the objects. They are the map between string fields and string values. Also, they are stored in attribute: value format, just like how the tables data is mapped to object using ActiveRecord in Rails. Small hashes are encoded in a very small space, so we should always try to represent our data using hashes.

And in this way using hashes our data parsing issue is solved. Now, we fetch data, as it is from Redis using Hashes, and there is no conversion of data format is involved.

Also memory consumption, reading and writing performance can be improved using optimized storage of hashes over strings data type.

Now lets check the above theory using Benchmark in rails. Here, we are going to use  redis-namespace and redis service which is explained later in this section.

Setting Data in Redis:

Benchmark.bm do |x|
  #here data is in the json format

  #Setting data using hash(value will be stored as hash)
  x.report { RedisService.new(klass: Event).set_list(key: CMS_MOVIE_LIST, data: data) }

  #Setting data using string(value will be stored as string)
  x.report { RedisService.new(klass: Event).set(key: MOVIE_LIST, data: data) }
end

#user     system   total    real
#0.030000 0.010000 0.040000 ( 0.011480) #Hashes
#0.150000 0.000000 1.150000 ( 0.447619) #Strings

Fetching Data from Redis:

Benchmark.bm do |x|

  #Fetching data using hash(value will be stored as hash)
  x.report { RedisService.new(klass: Event).get_list(key: CMS_MOVIE_LIST) }

  #Fetching data using string(value will be stored as string)
  x.report { RedisService.new(klass: Event).get(key: MOVIE_LIST) }
end

#user     system   total    real
#0.010000 0.000000 0.010000 ( 0.008200) #Hashes
#0.090000 0.000000 0.090000 ( 0.032398) #Strings

This demonstrates, how our performance can be improved by using Hashes over Strings in Redis.

So in order to use the same things in our rails application, we are going to use redis-namespace. For detailed information about this, refer Redis::Namespace

Initializing Redis in Rails

We instruct our rails app to use redis as a cache store and set the redis_host in ENV variable like this:


REDIS_HOST: 'redis://localhost:6379'

Now, initialize a wrapper around redis using redis-namespace. Or have a service redis_service.rb using redis-namespace so that we can interact with our redis.


class RedisService
  def initialize(klass:)
    redis = Redis.new(url: ENV['REDIS_HOST'], timeout: 1)
    @namespaced_redis = Redis::Namespace.new(klass, redis: redis)
  end

  def set(key:, data:, expire: nil)
    #Command to Set value of a key
    @namespaced_redis.set(key, data.to_json)

    #Expire your redis key In 1 week
    @namespaced_redis.expire(key, 1.weeks)
  end

  def set_list(key:, data:, expire: nil)
    #Command to Set List of Data on a Redis Key
    @namespaced_redis.set(key, Marshal.dump(data))
  end

  def get(key:)
    #Command to Get only Value of Key in Json Format
    JSON.parse(@namespaced_redis.get(key))
  end

  def get_list(key:)
    #Command to Get List of Data from Redis
    Marshal.load(@namespaced_redis.get(key))
  end

  def del(key:)
    #Command to delete Key from Redis
    @namespaced_redis.del(key)
  end

  def keys(pattern: nil)
    @namespaced_redis.keys(pattern)
  end
end

Marshal

In the above code, we are using marshal. It is a library in ruby which converts the collection of Ruby objects into byte stream. It is the fastest option available in ruby for data serialization. For detailed information refer this

Now we have generic Redis Service which we can use to perform different operations like add, delete, fetch data from Redis in our rails application.

Advantages of writing this service class:

  •  Code is DRY
  • All the redis commands are there in it, and we can use them whenever and wherever we want in our rails app.

Now, we are going to use this, to fetch movies on the basis of city.

Managing Redis Cache in Rails

Here, the whole idea is that, when a customer wants list of movies in a particular city, firstly we are going to fetch the movies, by directly quering on database. Secondly, we will cache the response using redis-namespace wrapper, such that on subsequent quering, the data will be fetched from redis, and not from the Database, thus improving our application performance.


class MoviesController &lt; ApplicationController
  #Here we are going to use the RedisService to perform operations on redis
  def index
    #Check if the list of movie is there in redis
    movies = RedisService.new(klass: Movie).get_list(key: "movies:#{params[:city]}")

    #If there is no movies in redis
    if movies.blank?
      #Load Movies from Database
      load_movies

      #serialize the data
      movies = serialize_resource(movies, V1::MoviesSerializer)

      #Cache the serialized response in Redis, so that it can be used again
      RedisService.new(klass: Movie).set_list(key: "movies:#{params[:city]}", data: movies, expire: 1.day)
    end

    #Returns the response
    mobile_success_response(data: movies)
  end
end

The above code is perfect, but there is one loophole in that, if any movie is added or it is updated in the database, it will not be shown to the customer if the data is fetched from Redis.

So in order to solve the above issue, what we have to do ?

We’ll write a callback in such a way that, whenever any movie is added or updated,  we will delete the keys corresponding to movie list. So, during updation of any movie, if User wants the data, it will be fetched directly from database and then will be stored in redis cache. On the subsequent calls, it will be fetched from redis. Below is the callback, to achieve this:


class Movie < ApplicationRecord
  after_commit :update_in_redis, on: [:create, :update]
  after_commit :delete_from_redis, on: [:destroy]

  def update_in_redis
    redis = RedisService.new(klass: self.class)

    #Delete all the keys matching the movies: pattern
    redis.del(key: redis.keys(pattern: "movies:*"))
  end

  def delete_from_redis
    redis = RedisService.new(klass: self.class)

    #Delete a movie from redis if it is deleted from database
    redis.del(key: self.id)
  end
end

Hope this blog will be useful. For more information like this, Stay tuned 🙂

Feature Testing Using Capybara

Feature testing is high level testing that allows us to go through our entire system, ensuring that each and every component is working perfectly fine. The test-cases written for it are termed as feature specs, which requires capybara.

In this blog, we will be focussing on writing feature specs using capybara with rspec. More specifically, we are going to look at capybara key features as well as, how capybara specs looks like.

We as a developer are mainly focussed about the underlying things, but we should also consider end user’s view. So, Capybara comes to rescue, it lets you write yours test-cases from user’s perceptive i.e, how end users would interact with your system. It is a ruby gem build on the top of underlying web-based driver and these drivers drive our browsers based on what our master ‘capybara’ says.

There are multiple web-drivers supported by capybara, some of these are as follows:

  • rack:test

    Capybara supports this web-driver by default. It is a library that sits above the rack app, and allows to fake web-requests to the app as a result making it considerably faster than other web-drivers because in reality we haven’t started a server. But it lacks JavaScript support.

  • selenium-webdriver

    It is one of the most popular driver and by default it supports Firefox browser where we can actually visualize our tests. It uses browser’s Javascipt Engine making it feel like having a actual human Q/A department interacting with our system. Also, we can configure it to be used in headless-mode which is specifically good from time and CI perspective.

  • capybara-webkit

    This one is comparatively faster than selenium-webdriver as it doesn’t load our entire app and also supports JavaScript, making it useful for true headless testing. It depends on Qt, a cross platform development toolkit. For more information, you can refer this capybara-webkit and Qt.

  • poltergeist

    This one runs our test-cases in PhantomJS browser with full JavaScipt support. It decreases the overall testing time by reducing the number of dependencies needed by CI server. But to use this, our system must have PhantomJS installed. Also, TravisCI, CircleCI have PhantomJS preinstalled. For more information you can refer poltergeist and phantomjs.

Setup

Provided you have installed Ruby, Rails and Rspec, we have to install Capybara. Here we will be using poltergeist driver which depends on PhantomJS, so firstly we need to have PhantomJS.

  • Installing PhantomJS

    Homebrew::

     brew install phantomjs
    

    MacPorts::

     sudo port install phantomjs
    
  • Installing Capybara and Poltergeist

    Add this to Gemfile and run bundle install.

     #Gemfile
    
     ...
    
     gem 'capybara'
     gem 'poltergeist'
    

    After installing it, we need to tell our RSpec that we want to use Capybara with Poltergeist by adding this to spec/rails_helper.rb:

     #spec/rails_helper.rb
     
     ....
     
     # load up Capybara
     require 'capybara/rspec'
     require 'capybara/rails'
    
     # load up Poltergeist
     require 'capybara/poltergeist'
     #Capybara.javascript_driver = :other_web_driver
     Capybara.javascript_driver = :poltergeist  
    

    If we want to use web-driver other than poltergeist, we can include that here in place of poltergeist in the above mentioned file. But if we do not specify any capybara driver, than by default it will use rack::test.

Application Example

As we have already installed everything, lets move onto how to write feature specs using capybara?

Consider a scenario, where our app has a home page and on that page we have a Log in button, so when a already registered user clicks on that Log in button she/he should be taken to a dashboard page.

Let’s test this feature using capybara:

require 'rails_helper'

RSpec.feature "Log In" do
  background do
    @user = create :user, email: 'test@gmail.com'
  end

  scenario "signing in with valid credentials", js: true do
    visit '/homes'
    
    expect(page).to have_link('Log in')
    
    within(".login") do
      click_link 'Log in'
    end

    within(".sign-up") do
      fill_in 'Email', with: @user.email
      fill_in 'Password', with: 'testpassword'
      click_button 'Log In'
    end
    expect(page).to have_text('Dashboard page')
  end
end

Here, we have used js:true to tell capybara that it has to use JavaScript driver for this test-case, otherwise if we don’t mention, it will by default use rack::test

One of the problem that I have faced when using Capybara with Rspec is that sometimes my test-cases were passing, while other times they were failing. So after searching I found that the main cause was because when we have feature spec with Capybara’s JavaScript driver, in that case the app-under-test and specs does not share a database connection. So, the solution is to use DatabaseCleaner truncation strategy instead of transaction one.

In order to use DatabaseCleaner, we have to include this in our Gemfile and run bundle install:

#Gemfile

....

group :test do
 gem 'database_cleaner'
end

After that, we have to configure our DatabaseCleaner strategy in spec/rails_helper.rb:

#spec/rails_helper.rb
RSpec.configure do |config|
  config.use_transactional_fixtures = false

  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.around(:each) do |example|
    DatabaseCleaner.strategy = :truncation

    # Start transaction
    DatabaseCleaner.cleaning do

      # Run example
      example.run
    end
  end
end

There are many ways as well as frameworks for doing end-to-end testing, one is using capybara with rspec which we have already covered in this blog. We can also do this testing using minitest that I’ll be covering in my next blog. Stay tuned 🙂