Core Data Tutorial: How To Preload/Import Existing Data

Failed Banks Preloaded Data with Core Data

This is the second part of a three part series to help get you up to speed with the basics of Core Data quickly.

In the first part of the series, we created a visual data model for our objects, ran a quick and dirty test to make sure it works, and hooked it up to a table view so we could see a list of our objects.

In this part of the series, we’re going to discuss how to import or preload existing data into Core Data so that we have some good default data when our app starts up.

In the final part of the series, we’re going to discuss how we can optimize our app by using NSFetchedResultsController, to reduce memory overhead and improve response time.

Preloading / Importing Existing Data

So how do we preload data into a Core Data store, anyway? Well, there are two popular solutions to this problem:

• Fill in Core Data on startup from external source. For this the app can start up, notice that the database hasn’t been imported yet, and start reading in data from an external source (such as an SQLite database or XML file) and then start inserting the data into Core Data.
• Provide pre-filled in SQLite database. For this we’ll let Core Data create the database structure for us based on the model, and then we populate the database with a utility app. The utility app could be a Mac or iPhone app that uses Core Data to populate the database via Core Data APIs, or some kind of program that fills in the SQLite database directly. Once the database is populated, just include it with the app and make the app use it as the default database if no database already exists.

We’re going to go with option 2 because it’s simple and more efficient. To populate the database, we’ll just extend our Python script a bit since we have that working already.

Note that by using a Python script to import the data rather than working with a utility app that uses Core Data APIs, it’s more likely to break in the future because we’re kind of going under the hood here… but for this tutorial I thought a) it’s a better learning experience since we just covered SQLite and it shows how things are working more clearly, and b) it is simpler!

So, let’s grab a copy of the sqlite database that was generated by our project. The easiest way to find the file is to set a breakpoint inside the application delegate, inside the persistentStoreCoordinator function, below the storeUrl line. You can examine the storeUrl variable to see a full path to where the sqlite backing file resides. So find this and copy it to the directory your Python script is in.

Once you have the database, use sqlite3 to take a peek at how the database looks:

sqlite3 FailedBanksCD.sqlite
sqlite3> .schema
CREATE TABLE ZFAILEDBANKDETAILS ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
Z_OPT INTEGER, ZZIP INTEGER, ZINFO INTEGER, ZUPDATEDDATE TIMESTAMP,
ZCLOSEDATE TIMESTAMP );
CREATE TABLE ZFAILEDBANKINFO ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
Z_OPT INTEGER, ZDETAILS INTEGER, ZNAME VARCHAR, ZSTATE VARCHAR,
ZCITY VARCHAR );
CREATE TABLE Z_METADATA (Z_VERSION INTEGER PRIMARY KEY,
Z_UUID VARCHAR(255), Z_PLIST BLOB);
CREATE TABLE Z_PRIMARYKEY (Z_ENT INTEGER PRIMARY KEY, Z_NAME VARCHAR,
Z_SUPER INTEGER, Z_MAX INTEGER);
CREATE INDEX ZFAILEDBANKDETAILS_ZINFO_INDEX ON ZFAILEDBANKDETAILS (ZINFO);
CREATE INDEX ZFAILEDBANKINFO_ZDETAILS_INDEX ON ZFAILEDBANKINFO (ZDETAILS);
sqlite> select * from ZFAILEDBANKINFO;
1|2|1|1|Test Bank|Testland|Testville
2|2|1|2|Test Bank|Testland|Testville
3|2|1|3|Test Bank|Testland|Testville
sqlite> select * from ZFAILEDBANKDETAILS;
1|1|1|12345|1|292794835.855615|292794835.679693
2|1|1|12345|2|292794875.943392|292794875.768675
3|1|1|12345|3|292795809.375025|292795809.215297
sqlite> select * from Z_PRIMARYKEY;
1|FailedBankDetails|0|3
2|FailedBankInfo|0|3

Here’s a quick description of what’s in here. Z_METADATA contains some information about the Model that Core Data needs to function. Z_PRIMARYKEY contains (among other things) information on the max key that is currently used by each entity.

As for ZFAILEDBANKINFO and ZFAILEDBANKDETAILS, those are our main data tables! Z_PK is the unique id for each, Z_ENT is their entity id (same as what’s listed in the Z_PRIMARYKEY table), and finally there are our normal fields.

Now, let’s create a Python script to populate this database by reading in the contents of our old database, and creating the appropriate rows in the new database. Create a Python script like the following:

import sqlite3;
            from datetime import datetime, date;
            import time
             
            inConn = sqlite3.connect('banklist.sqlite3')
            outConn = sqlite3.connect('FailedBanksCD.sqlite')
             
            inCursor = inConn.cursor()
            outCursor = outConn.cursor()
             
            outConn.execute("DELETE FROM ZFAILEDBANKINFO")
            outConn.execute("DELETE FROM ZFAILEDBANKDETAILS")
             
            maxId = 0
            inCursor.execute("select * from failed_banks")
            for row in inCursor:
            closeDate = datetime.strptime(row[5], "%Y-%m-%d %H:%M:%S")
            updatedDate = datetime.strptime(row[6], "%Y-%m-%d %H:%M:%S")
            closeDateSecs = time.mktime(closeDate.timetuple())
            updatedDateSecs = time.mktime(updatedDate.timetuple())
             
            # Convert time references secs to NSDate reference
            deltaSecs = time.mktime((2001, 1, 1, 0, 0, 0, 0, 0, 0))
            closeDateSecs = closeDateSecs - deltaSecs
            updatedDateSecs = updatedDateSecs - deltaSecs
             
            if row[0] > maxId:
            maxId = row[0]
             
            # Create ZFAILEDBANKINFO entry
            vals = []
            vals.append(row[0]) # Z_PK
            vals.append(2) # Z_ENT
            vals.append(1) # Z_OPT
            vals.append(row[0]) # ZDETAILS
            vals.append(row[1]) # ZNAME
            vals.append(row[3]) # ZSTATE
            vals.append(row[2]) # ZCITY
            outConn.execute("insert into ZFAILEDBANKINFO values(?, ?, ?, ?, ?, ?, ?)", vals)
             
            # Create ZFAILEDBANKDETAILS entry
            vals = []
            vals.append(row[0]) # Z_PK
            vals.append(1) # Z_ENT
            vals.append(1) # Z_OPT
            vals.append(row[4]) # ZZIP
            vals.append(row[0]) # ZINFO
            vals.append(closeDateSecs) # ZUPDATEDATE
            vals.append(updatedDateSecs) # ZCLOSEDATE
            outConn.execute("insert into ZFAILEDBANKDETAILS values(?, ?, ?, ?, ?, ?, ?)", vals)
             
            outConn.execute("update Z_PRIMARYKEY set Z_MAX=?", [maxId])
             
            outConn.commit()

This should be pretty straightforward. The only tricky thing we needed to do here was to convert the dates from the format they were stored in our old database (a string) to the format they are stored in the new databaes (epoch seconds). Update: Jeff R. pointed out that NSDate uses a difference reference date than Python does so we have to convert that – see the comments section for more information.

Give the Python script a whirl and if all works well you should be able to see your DB populated with data:

python coreData.py
sqlite3 FailedBanksCD.sqlite
sqlite> select * from ZFAILEDBANKINFO limit 3;
1|2|1|1|Desert Hills Bank|AZ|Phoenix
2|2|1|2|Unity National Bank|GA|Cartersville
3|2|1|3|Key West Bank|FL|Key West
sqlite> select * from ZFAILEDBANKDETAILS limit 3;
1|1|1|57060|1|1269576000|1269576000
2|1|1|34678|2|1269576000|1269576000
3|1|1|34684|3|1269576000|1269576000

Once you’re satisfied the data is in the database correctly, drag the populated database file into your XCode project in the Resources folder. Then open up FailedBanksCDAppDelegate.m, navigate to the persistentStoreCoordinator function, and replace the storeUrl line with the following:

NSString *storePath = [[self applicationDocumentsDirectory]
            stringByAppendingPathComponent: @"FailedBanksCD.sqlite"];
            NSURL *storeUrl = [NSURL fileURLWithPath:storePath];
             
            // Put down default db if it doesn't already exist
            NSFileManager *fileManager = [NSFileManager defaultManager];
            if (![fileManager fileExistsAtPath:storePath]) {
            NSString *defaultStorePath = [[NSBundle mainBundle]
            pathForResource:@"FailedBanksCD" ofType:@"sqlite"];
            if (defaultStorePath) {
            [fileManager copyItemAtPath:defaultStorePath toPath:storePath error:NULL];
            }
            }

All this does is check to see if the sqlite file already exists in the documents directory, and if it doesn’t (i.e. it’s the first time the app has run) it copies the included database from the bundle into the documents directory. Then Core Data will start using the preloaded data and we’re good to go!

Delete the old SQLITE3 file from the iPhone simulator directory (or click iPhone Simulator\Reset Contents and Settings to clear everything), and re-run your app. If all works, you should now see the prepopulated list of banks!

Where to Go From Here?

At this point, we have a detail view that works pretty much as efficiently as the way it did in our SQLite example. However, with Core Data, with just a couple more steps we can make it even more efficient. So next time we’re going to cover how to do that by using NSFetchedResultsController!