CouchDB and Ionic

CouchDB, PouchDB, and Ionic 2: An Introduction



·

Databases. The creatures that often live somewhere in the cloud and hoard all of the data for your application. Although they can be troublesome to work with, they allow you to turn your isolated application into one that can share and communicate with the world.

There are many different options available, and different options are going to make sense for different people and different scenarios, but I’ve spent a long time looking for the one that “felt right” – the one that would serve well as a good default when building web-based mobile applications. This is how I felt about Ionic for building the front end of mobile applications, and it’s how I feel about CouchDB for the back end of mobile applications.

CouchDB is a document based NoSQL database that can easily be synced to other CouchDB databases. If you are unfamiliar with what NoSQL is all about, and specifically if you are not familiar with document based NoSQL databases, I would recommend reading An Introduction to NoSQL for HTML5 Mobile Developers first.

There’s many advantages to using CouchDB including the ease of which it can be scaled, and the speed of read and write operations, but the killer feature when it comes to mobile applications (and the reason I think it makes a great default) is its ability to synchronise between multiple databases. Synchronising two or more databases can be astoundingly difficult, and in order to provide offline functionality in an application, we need to provide local data to the user that can be read and modified, and then later synced back to the main database or databases when a network connection becomes available.

When using CouchDB, it is trivially easy to do this. A local database can be synced to a remote database with PouchDB (which is compatible with any CouchDB database, and which we will often be using) with a single line of code:

PouchDB.sync('mydb', 'http://url/to/remote/database');

(you can also sync databases using CouchDB’s HTTP API directly)

All we need to do is supply the URL for the remote database. I’ve yet to come across another backend solution that so greatly simplifies such a complex task.

CouchDB even has an integrated web server with an HTTP API so you can interact with the database directly through its URL if you don’t want to include some kind of server middleware for your application (like Node or PHP).

CouchDB is simple and elegant, but it’s also strange and different. Initially, it’s going to be difficult to figure out how to build the things you are used to building (especially if you are used to relational databases like MySQL). Part of CouchDB’s philosophy is that it will make it hard for you to do things that won’t scale well.

I intend to create a series of tutorials on building Ionic 2 applications with CouchDB as the backend (I have already created tutorials on CouchDB in the past, but I’d like to do it more thoroughly), but first I want to cover a little theory. In this article, I will be introducing the general concept of CouchDB, how the data is structured, and how it is able to so effectively scale and provide effortless synchronisation features.

What’s PouchDB?

If you’re using CouchDB and you’re building mobile applications, then you will most definitely come across PouchDB at some point. PouchDB is a critical ally to CouchDB in the mobile world and will allow you to run a local instance of a CouchDB style database on the device that can automatically sync to a remote CouchDB database. This allows for effortless offline data synchronisation and we will most likely always use PouchDB locally when building an application with a CouchDB backend.

I will mostly just be talking about CouchDB in this article since I am just attempting to explain the basic concepts, but keep in mind that these two technologies will go hand in hand.

Data Structure

Data in CouchDB, like in many (but not all) NoSQL databases, is stored as documents and there is no pre-defined schema. Unlike a relational database where we need to describe what the structure of our data is before we insert it (a schema), and can only insert data that matches that schema, with CouchDB you can store whatever kind of data you want, whenever you want. This doesn’t mean that there is no structure to a CouchDB database, it is critically important that you add data in a way that makes sense and will perform well for your application. This creates a lot of flexibility for data storage, but it also has its downsides and it can be quite hard to learn because there is no “100% correct” way to store the data, and in many cases, it’s better to do things you will probably have learned to avoid like duplicating data.

A document in CouchDB is essentially just a JSON string that looks something like this:

{
    "_id": "00a271787f89c0ef2e10e88a0c0001f4",
    "_rev": "1-2628a75ac8c3abfffc8f6e30c9949fd6",
    "item": "apple",
    "prices": {
        "Fresh Mart": 1.59,
        "Price Max": 5.99,
        "Apples Express": 0.79
    }
}

This is essentially how all document based databases work, it’s just a collection of these JSON strings. They are called documents because that is an analogy for what they are supposed to be – a document like an invoice, an email, or a receipt that contains all of the relevant data in one place. We can immediately see in the document above the various prices for apples at different stores, but in a relational database these would likely be split up into different tables and joined using a foreign key when necessary.

A document, like an email, can also have “attachments”. An attachment is essentially just another document that is related to document it is attached to (a song might be attached to the information for that song).

The main difference in a CouchDB database, when compared to other document based databases, is that it also stores a “revision” or _rev field for each document, but we will discuss why that is relevant in the next section.

Consistency, Revisions, and Offline Syncing

Lack of consistency is the reason for most of the advantages CouchDB provides – easy scalability, fast reading and writing, and synchronisation.

Consistency refers to the ability to keep a set of data in sync, meaning that wherever or whenever you access the data it is going to be a true reflection of what that data should be, and it might seem a little concerning on the surface that CouchDB does not treat consistency as a priority but instead takes an “eventual consistency” approach – the data will be consistent eventually, but you can continue to use the database whilst it is not consistent.

In reality, this is a tradeoff. An RDMS like MySQL prioritises Consistency and Availability over Partition Tolerance, whereas CouchDB prioritises Partition Tolerance and Availability over Consistency.

So, what does this lack of consistency actually mean? To give an example, let’s imagine a scenario that we will likely encounter later. Let’s say that we have a todo list application where two users are both trying to edit the same todo. We’ve set up offline syncing so we have three databases in play here:

  1. User One’s local CouchDB database
  2. User Two’s local CouchDB database
  3. A remote CouchDB database

Thanks to CouchDB’s synchronisation features, all three of these databases are in sync and contain the same data. Let’s also assume that both users are currently offline so their changes won’t be replicated immediately to the main database. If both users attempt to edit the same todo, what happens?

CouchDB isn’t worried so much about consistency, so it will let both users update their local database with the changes (without checking what the state of the data is in the remote database). As of right now, the two users have different sets of data, so the data is in an inconsistent state. When the users eventually come back online their changes will be synced with the main database, but clearly, only one of the users updates can be used to update the todo.

To deal with this situation, CouchDB assigns documents with a “revision” number, which is stored in the _rev field that we saw in the document example earlier in the example. In order to update a document in CouchDB you must first retrieve the entire document, modify it in any way you wish, and then send the entire document back. When sending the document back it is important that you also send the _rev field, and that the _rev on your document matches the _rev on that same document in the remote database. If the _rev fields match, CouchDB will process the update and increment the _rev number. If the _rev fields do not match, the update will be rejected.

What this means for our situation with two simultaneous users, is that whoever syncs their update to the remote database first will “win” and have their update accepted, because their _rev will match the _rev stored in the remote database. However, when the second person attempts to sync their data, their _rev field will no longer match because it will have been incremented because of the “winners” update.

Even though the update will be rejected, it won’t just be completely lost. CouchDB will keep that document and mark it with a special property "_conflicts":true, so you would be able to decide how to handle this scenario in your own application (i.e. do you just want to just ignore this change, or would you prefer to notify the user and let them decide?).

The benefit of this approach is that we can do really quick reads and writes in the local database (we don’t have to worry about going through the network to access the database), and we can easily add multiple instances of our database without having to worry about how we are going to handle communication between them.

Scaling

When it comes to scaling any kind of application, things can get a little complicated. If you are using a MySQL database for example, you can’t just fire up another server and copy your MySQL database over to it, send half of your traffic to one and half to the other. It’s a complicated problem with no easy solution.

However, CouchDB is designed to scale easily. I am not going to get into scaling strategies in this tutorial, but I wanted to highlight that you can, more or less, just fire up some more CouchDB instances to help scale your application.

Summary

The immediately noticeable benefit to CouchDB is that you can easily create mobile applications that work offline and sync when online, and the added bonus down the track is that your application is going to be a lot easier to scale, and potentially a lot faster.

Over the following weeks, I will release a series of tutorials that will walk through the basic concepts (and perhaps some more advanced concepts as well) of building backends for Ionic applications with CouchDB.

What to watch next...

  • JB

    Hi @joshuamorony
    Very interesting article!

    what about security?

    For example, PouchDB.sync(‘mydb’, ‘http://url/to/remote/database’);
    Everyone would can trying sync my database and send me not wanted data.

    Is it so?

    Regards!

    • That’s definitely something I will be covering. I do already have an article up about using the “Database per user” model for security, but I will cover it more in depth.

    • Mike Ward

      JB, if you can’t wait for Josh’s next article, good place to start is here: https://github.com/nolanlawson/pouchdb-authentication#couchdb-authentication-recipes
      You can see how the PouchDB authentication plugin supports various different types of authentication, from pretty open to incredibly restrictive.
      Combining accounts, roles, multiple databases (e.g. one per user, or one per role) you should be able to make the exact security profile you need for your app.

  • Robson Curvello

    Hello Josh! Thanks for sharing your knowledge with us!… Well I dont know if you know the Strava app. this app use the same data in app and sync with the Webpage. How Can I do it? and I can open my user on web and use another features avaliable only by webpage. Is possible sync my app data with the mysql or another database based in SQL? Sorry if my question is a little confuse, and my english too. LOL

    • Couch/Pouch will allow you to sync data easily between multiple clients and allow for offline use. Using MySQL you could just have both your website and app pointing to the same server to load the same data, but if you wanted to replicate/sync that data for offline use it would be a lot harder (support for this type of behaviour isn’t built in like it is with Couch/Pouch)

  • I would love to hear your opinions on when CouchDB might be a better option over Firebase.

    • I don’t really know either of those databases well enough to make any insightful comparison – I have used Firebase before but only very lightly and a long time ago. From the surface, it seems to me that the major difference is CouchDB’s ability to sync/replicate databases which allows for offline functionality. CouchDB’s revision system would also come in useful for certain types of applications. I think a lot would come down to preference as well, CouchDB’s method for retrieving data by using MapReduce might not be everybody’s cup of tea (e.g. query ability like that in MongoDB I has less of a learning curve).

      • I’m no Firebase expert either, but I do think the revision system is a differentiator.

        Firebase has offline capabilities, but you have to use the iOS/Android SDK to get that. (This is why I’m surprised so many Ionic+Firebase tutorials using the JS SDK, except that maybe its good for a quick prototype or MVP app.)

        One thing that becomes super important when/if your app starts to see some traction is how easily you can access the data for ad-hoc queries, reporting, etc. You need those things for analytics, troubleshooting, and customer support.

      • The next article I’ll be releasing (next week) will be covering how to create MapReduce views to query data, I actually quite like this method for retrieving data and it’s quite easy to write a MapReduce function that would say return the number of customers with an abandoned cart in the last week. You might create a view of all abandoned carts using the date as the key, then you can easily query that view for abandoned carts within any date range.

    • Mike Ward

      I had a close look at both Firebase and CouchDB for my current project.
      One thing that made a big difference for me, was if you’re planning on targeting users in China, Firebase is blocked. So, without some kind of proxying functionality to skirt the Firewall, you’re in trouble.
      I’m running some CouchDB servers on DigitalOcean at the moment and they seem to be working very well from everywhere.

  • NT

    Hi, we use ionic and pouchDB in iOS, WinPho, Android.
    All work fine apart from in Apple App Store Test where we see server logs for the api calls, but not for the couch calls.
    So
    https://api.example.com connects OK
    https://couch.example.com doesn’t connect

    Have you see this ? It started with iOS 10. We adjusted the CSP, it worked. Now it’s stopped again.
    Here’s the CSP:

    Odd thing is it works fine everywhere except in Apple’s test lab (Cupertino).

    • Mike Ward

      Are you using WkWebView? I’ve had some (still unresolved) issues with CouchDB recently with WkWebView – the same thing you’re reporting. I fiddled with countless CSPs and never made progress.
      It’s sitting in my issue log for the time being waiting for a much more thorough investigation!

      • NT

        So, turns out it was a timing issue (is it ever thus !!). There was another piece of code inside a Javascript promise that with the right timing would try to log in with incomplete creds, knocking out the session created by another dialog.
        Cupertino was in a network connection speed window that could only be reproduced using the [Apple Network Link Conditioner][1].

        We live and learn !!

        [1]: http://stackoverflow.com/questions/9659382/installing-apples-network-link-conditioner-tool

      • Mike Ward

        Yes, you’re right. It’s ALWAYS timing one way or another, isn’t it? 🙂

        Ooo – the Apple Network Link Conditioner looks very useful! Didn’t know about that.
        Thanks for sharing 🙂

  • Alexandre Bueno

    Hello Josh.
    Thank you for the tutorials.
    Could you please post a example with image attachment with pouchDB, couchDB and Ionic2?

    Thank You very much

  • franzz2000

    Hi Josh,

    great post and great explanation. That was what I was looking for. I really look forward to read the new posts about this subject. Hope it will be soon.

    Franz

  • amine